Delete lines beginning with X to ending with Y

Robert Oram · May 18, 2023, 4:38 PM

Hi,
I have log files with some lines I want to remove either with a Regex or a Macro.
I need to delete lines, beginning with “ATTENTION:”, to the line “Backlog (Last messages accounting for this alert)”.

Any suggestions are appreciated!
Thanks,
Bob

Raw data:

{index=vpn-auth_12, message=May 13 2023 09:28:31:
ATTENTION: The email came from an external source. 
 
Event Definition
Title	VPN-Report-test
Description	
Type	aggregation-v1

Event
Timestamp	2023-05-14T09:30:36.057-04:00
Message	VPN-Report-test: count(facility_num)=977.0
Source	arch-syslog01
Key	
Priority	2
Alert	true
Timestamp Processing	2023-05-14T09:30:36.057-04:00
Timerange Start	2023-05-13T09:30:36.057-04:00
Timerange End	2023-05-14T09:30:36.057-04:00
Source Streams	6303d45e9c80f55bfa35b2df
Fields	

Backlog (Last messages accounting for this alert)
{index=vpn-auth_12, message=May 13 2023 09:31:52:

Final result:

{index=vpn-auth_12, message=May 13 2023 09:28:31:
{index=vpn-auth_12, message=May 13 2023 09:31:52:

—

moderator added code markdown around text; please don’t forget to use the </> button to mark example text as “code” so that characters don’t get changed by the forum

PeterJones · May 18, 2023, 4:43 PM

@Robert-Oram ,

Use ^ to anchor to the beginning of the line, and use . matches newline or the regex (?s) equivalent to make .*? go across multiple lines. Also, because parentheses are meaningful to regex, you will need to escape them as \( and \)

For example, FIND = (?s)^ATTENTION:.*Backlog \(Last messages accounting for this alert\)\R and REPLACE WITH nothing, in regular expression mode, gave the Final Result you asked for.

Here’s a screenshot with the match and dialog shown:

----

Useful References

guy038 · May 18, 2023, 5:07 PM

Hello @robert-oram, @peterjones and All,

So, from your INPUT text :

{index=vpn-auth_12, message=May 13 2023 09:28:31:
ATTENTION: The email came from an external source. 
 
Event Definition
Title	VPN-Report-test
Description	
Type	aggregation-v1

Event
Timestamp	2023-05-14T09:30:36.057-04:00
Message	VPN-Report-test: count(facility_num)=977.0
Source	arch-syslog01
Key	
Priority	2
Alert	true
Timestamp Processing	2023-05-14T09:30:36.057-04:00
Timerange Start	2023-05-13T09:30:36.057-04:00
Timerange End	2023-05-14T09:30:36.057-04:00
Source Streams	6303d45e9c80f55bfa35b2df
Fields	

Backlog (Last messages accounting for this alert)
{index=vpn-auth_12, message=May 13 2023 09:31:52:

Two solutions are possible :

The @peterjones’s idea, which deletes anything between the line begining with the ATTENTION: string and ending at the complete end of the line beginning with the Backlog string

Thus :

SEARCH (?xs-i) ^ ATTENTION: .*? ^ Backlog \x20 .+? \R
REPLACE Leave EMPTY

Secondly, use the following regex S/R which deletes any line which does not begin with the {index= string :

SEARCH (?x-si) ^ (?! {index= ) .+ \R | ^ \R
REPLACE Leave EMPTY

Whatever the option chosen, you’ve left with this OUTPUT text :

{index=vpn-auth_12, message=May 13 2023 09:28:31:
{index=vpn-auth_12, message=May 13 2023 09:31:52:

Best Regards,

guy038

Robert Oram · May 18, 2023, 5:52 PM

Thank you so much for your prompt help! They both work perfectly!

Bob