Users of security log data are collecting more needles than ever — but they still feel as though they must comb the haystack to find them, according to a study released yesterday.

According to the “SANS 2011 Log Management Survey” (PDF) published this week, organizations are collecting and logging more security data than ever before. But they still have trouble finding and analyzing the specific information they need to diagnose potential security problems.

“When this survey started seven years ago, log collection was only being done by 43 percent of respondents, compared with 89 percent who indicated they collected logs this year,” the report states. “Now, [enterprises are] also collecting logs for much more than detecting suspicious behavior and troubleshooting, as in the recent past. Over the past two years, more respondents are also collecting logs for use in forensic analysis and correlation and to meet/prove regulatory compliance.”

Organizations are collecting more types of log data than ever before, adding physical systems such as HVAC and SCADA to the mix, the report says. “This means more log types to collect and analyze—each with their own data formats that can vary widely,” the report says.

“The mechanics of collecting, storing and archiving the log data are no longer the challenge in today’s world of almost unlimited data storage,” SANS says in the report. “The challenge now is extracting the needed information for monitoring, management, compliance and decision-making from what respondents say is upwards of 100,000 events recorded per day.”

Real-time alerts are the most useful feature of log management tools, according to respondents. “However, they were less enthusiastic about their log management system’s ability to interface with third-party tools or larger SIEM environments,” the report states. “Users also cited problems with correlation, searching and interfacing with heterogeneous systems, and difficulties locating information within logs.”

Read more: DarkReading