Open-source intelligence (OSINT) involves gathering information from public sources to achieve various objectives, whether for IT security, detecting malicious activities, or collecting information for intelligence operations. Advanced techniques for OSINT help analysts sift through abundant data to find specific information.
OSINT tools help access, collate, and organize relevant data based on queries within the various tools. Organizations wanting to develop an OSINT practice should consider what approach they should take when leveraging these tools.
The vast amount of data available on the Internet is publicly accessible. Organizations leverage the Internet to help communicate their companies' following products and services, their financial earnings report and key customer wins. Currently, 66% of the world's population uses the Internet daily. With this access comes the additional data.
According to Exploding Topics, data continues to increase on the internet daily at a rate most will find hard to fathom.
Much open-source data is bound to news, media, social media profiles, company websites, learning institutions, blogs, Wiki posts, and online publications.
Public information regarding previous cybersecurity threats, including data breaches, insider threats, and other criminal activities, is a standard search content piece for OSINT. Organizations can learn a great deal by studying previous attacks against other organizations.
Many industries search, process, and rationalize open-source data. The modern-day news cycle requires journalists to be first with the big story. Often, a big tale starts with scattered information across several unconnected resources.
OSINT is ideal for journalists to use to capture story fragments and create a narrative based on the open-source information captured. Ultimately, this narrative becomes the lead news clip or the foundation for the blog.
Like journalism, cybersecurity benefits from accessing source-gathering tools using advanced search techniques. Gathering threats from previous cyberattacks, messages embedded within social media sites forecasting upcoming attacks are valuable snippets.
Cybersecurity devices generate a significant amount of data. Most security telemetry data will not be released to the public. However, security vendors will provide deeper insight into breaches of the source code embedded within their products and recommend patching their solutions. Public data analysis tools provide the means to capture and identify critical notifications from security vendors and the patch release timeline. With nearly 20,000 security vendors globally, OSINT tools will help set a priority flag as part of their workflow.
Law enforcement also captures similar cybersecurity alerts as part of their investigations. Many law enforcement agencies capture open-source content through various online investigation tools, either specific to a criminal they are investigating or an organization under suspicion. News articles, publications, and podcasts are excellent sources of content law enforcement can add as artifacts to their cases. Leveraging OSINT tools helps expedite the data-capturing process more than just using a browser search page.
Germany now has a federal open-source policy focusing on digital sovereignty. The Centre for Digital Sovereignty of Public Administration (ZenDiS) was created in 2022 to support this initiative.
The coordination of open-source initiatives between state and federal levels is still evolving. German cities are expressing interest in utilizing open source despite potential challenges in coordinating with federal governance. As the landscape evolves, federal policies are increasingly promoting the use of open source in all public administrations.
Individuals and organizations wanting to query open-source data leverage standard search engines like Google, Baidu, and Bing. These search engines provide basic search functionality and results-based search engine scoring.
Discovery tools analyze and uncover extensive information. Google, often seen as a standard search engine, can provide valuable insights in the hands of an OSINT expert and cybersecurity analyst.
Scraping tools help gather and filter data from websites efficiently and securely. They ensure the removal of specific data while minimizing detection and eliminating unnecessary data interference.
Aggregation tools simplify data processing, securely analyze information, and provide actionable insights by connecting data fragments and presenting them in an easy-to-understand format.
Many tools are available for gathering OSINT data, including the OSINT Framework. The OSINT Framework is a free tool maintained by Justin Nordine that helps with advanced Google searches to gather information.
“Maltego automates searches from public data sources, allowing users to execute multiple queries with one click. It can transform actions or search plans and access public information sources like DNS records and social networks. The program can easily add new searches or create custom ones because it is compatible with almost any source with a public interface.”
This tool finds connections between people, companies, and online information and presents them in clear charts. Each graph can have up to 10,000 data points.
“Maltego offers a free version with limited features, known as Maltego CE. The desktop version, Maltego XL, costs $1,999 per instance. Server installations for large-scale commercial use start at $40,000 and include a complete training program.”
Mitaka is a versatile tool that can function as a Chrome extension or Firefox add-on. It allows users to search multiple search engines for online indicators like IP addresses, domains, URLs, and hashes.
This tool helps identify malware, sketchy sites, and shady emails to increase awareness and security.
SpiderFoot is a free tool for gathering and analyzing information from various sources. This tool gathers data on IP addresses, domains, and emails. The tool is available on GitHub, with a command-line interface and a web-based GUI.
Spyse provides complete DAAS solutions for Internet security professionals, corporate administrators, SSL/TLS certificate providers, data centers, and business analysts.
“A free plan is offered, but developers using the Sypse API for app development may need to consider purchasing a paid subscription.”
“Intelligence X is an archival service and search engine that preserves historical versions of web pages and removes leaked data sets from the web for objectionable content or legal reasons.
Interesting Fact: “The service has indexed data from the email servers of prominent political figures such as Hillary Clinton and Donald Trump, footage from the 2021 Capitol Hill riots, and Facebook's data leak of 533 million profiles. This information can be valuable to intel gatherers, political analysts, news reporters, and security researchers.”
“For those new to the dark web, DarkSearch.ai is a helpful research platform. It is free to use and has a free API for automated searches. You can access DarkSearch.Ai from a web browser; Tor is unnecessary.”
Shodan is a search engine designed to find information on devices that are not easily searchable, including those within the Internet of Things (IoT). Other OSINT tools like the Harvester use Shodan as a data source.
“A Freelancer license allows anyone to scan up to 5,120 IP addresses monthly on Shodan for $59. A Corporate license provides unlimited results and scanning of up to 300,000 IPs monthly for serious users for $899. The Corporate version also includes a vulnerability search filter and premium support.”
Babel X is a search tool for the public internet that can search blogs, social media, message boards, news sites, and even the dark web, including Onion sites and some deep web content. It can geo-locate sources and perform text analysis to provide relevant results. Babel X can search in over 200 languages.
Multilingual search can be beneficial in various scenarios, such as staying informed about global news trends, identifying intellectual property on foreign websites, detecting potential compromises of critical partners, and uncovering user handles of suspected attackers on non-English message boards.
The information discovered by Metagoofil could be valuable to a hacker for purposes such as conducting brute-force password attacks or phishing schemes. Organizations can use the same OSINT data to safeguard or conceal it before malicious exploitation occurs.
Metagoofil, a tool available on GitHub, extracts metadata from public documents such as .pdf, .doc, .ppt, .xls, and more. It can investigate various types of documents accessible through public channels.
theHarvester is a tool for gathering public information outside of a network, making it useful for reconnaissance before penetration testing.
“This tool uses various search engines, including Bing, Google, dogpile, DNSdumpster, Exalead, Netcraft Data Mining, and AlienVault Open Threat Exchange. It can also access the Shodan search engine to find open ports on hosts. The tool collects emails, names, subdomains, IPs, and URLs.”
theHarvester is available on GitHub for anyone to download. When cloning the tool, it is advisable to set up a separate Python environment in a virtual environment.
Search code is a specialized search engine that delves deep into source code to gather intelligence for OSINT purposes. The engine, developed by a single individual, offers powerful search capabilities.
Developed with common attributes like ChatGPT, DarkGPT simply became a jailbroken version of OpenAI's tool. ChatGPT and its various large language models (LLMs) have various guardrails incorporated into their framework.
These guardrails only allow the processing of open-source data instead of copyrighted content from published authors. ChatGPT also has restrictions around the queries prompt engineers can leverage. DarkGPT removes these guardrails and more.
Having the ability to generate rogue hacking code to execute a data breach, DOS attack, or a video game based on illegal activities is at the heart of DarkGPT.
DarkGPT empowered the user to access several open-source data and private content embedded within the public domain. DarkGPT can now query content restricted to ChatGPT, including dark art and dark aspects of human existence, and ask pointed questions about the dark side of a writer, director, or politician.
Attackers often use OSINT to execute a social engineering attack on employees from social media platforms or using email spear-phishing attacks. LinkedIn and other social networking sites are valuable sources of information for attackers, as they reveal personal details that can be used for phishing and password guessing.
Content shared on social media and professional networks may seem harmless at first. However, cyber attackers can exploit this information alongside existing vulnerabilities to launch cyber attacks.
Using cloud resources, attackers can scan for vulnerable assets, open ports, and misconfigured data stores. They can also find credentials and other information on GitHub, where developers sometimes unknowingly share sensitive data in their code.
While open-source intelligence tools (OSINT) provide valuable insights and data from public sources, their optimal utilization requires expertise and specialized knowledge. These tools offer a vast amount of information that can be harnessed for various purposes, including cybersecurity, journalism, and law enforcement. However, the complexity of data collection, analysis, and interpretation necessitates skilled professionals to derive meaningful conclusions and actionable insights.
NovaMDR, offered by Forenova, plays a crucial role in this context by providing managed detection and response services that bridge the gap between open-source tools and expert analysis. NovaMDR's team of cybersecurity experts offers real-time monitoring, threat detection, and incident response within a short timeframe, ensuring comprehensive network visibility and proactive security measures. By leveraging NovaMDR's expertise, SMEs can effectively navigate the complexities of open-source intelligence tools and enhance their cybersecurity posture with confidence.
In essence, while open-source tools serve as powerful resources, the expertise and guidance provided by NovaMDR ensure that organizations can effectively harness the potential of these tools to bolster their security defenses and make informed decisions based on reliable intelligence.