So you forgot to set up logrotate on an active log eh? You’ve got a many gigabyte file to weed through and you need to extract a chunk of time from it?
Here’s a quick cheat sheet to help you get by, quickly and sanely.
It’s about byte offsets!
- Get the byte offset in the file where your time range starts
- Get the byte offset in the file where your time range ends
- dd the data out!
|
1 |
# Get first byte offset, leftmost number is the offset, "offset_start"<br />grep -m 1 -b "2011-11-15 11:3" error_log<br /> # Get last byte offset, "offset_end"<br />grep -m 1 -b "2011-11-16 01:3" error_log<br /><br /> #Subtract offset_start from offset_end to get byte length, then do:<br />dd if=error_log of=filtered bs=c skip=[offset_start] count=[byte_length]<br /> |
Caveats
- You should tack on extra bytes to the byte length, because the offset_end number is actually the beginning byte of your boundary log entry
- Figuring out the boundary is a bit tricky because a log entry -has- to be present in order to match, so if you’re looking for what happened at 20:00 hours on X date, you may have to round up to the date level depending on how busy your log is
- This is just a trick to extract a chunk of entries to speed up further filtering.
Full example
|
1 |
dev:/var/log/apache2# grep -m 1 -b "23/Nov/2011:00" access.log.1<br />6633711:3694 127.0.0.1:53784 - - 176us [23/Nov/2011:00:00:10 -0600]<br /><br />dev:/var/log/apache2# grep -m 1 -b "24/Nov/2011:19" access.log.1<br />9674209:10872 127.0.0.1:46097 - - 173us [24/Nov/2011:19:00:10 -0600]<br /><br /># byte length: 9674209 - 6633711 = 3040498 bytes (Plus 256 to cover the caveat) = <br />dd if=access.log.1 of=filtered bs=c skip=6633711 count=3040754<br /> |
