I had a nice chat with my buddy Andre today on some open issues with phpLogCon v2. With his permission, I am posting it here. Others may find it useful (or may not ;)). If you’d like to voice your opinion, please simply do…
Rainer Gerhards [15:10]:
on the windows event log format inside syslog messages
Rainer Gerhards [15:10]:
we could go with the syslog-protocol format: ftp://ftp.rfc-editor.org/in-notes/internet-drafts/draft-ietf-syslog-protocol-23.txt
Rainer Gerhards [15:10]:
see section 6.3.5 – samples
Andre Lorbach [15:11]:
ok one second
Rainer Gerhards [15:12]:
I had this use case on my mind what I crafted the draft
Andre Lorbach [15:12]:
loooks good to me
Rainer Gerhards [15:12]:
an elaborate sample a bit later on page 19
Rainer Gerhards [15:12]:
1 2003-10-11T22:14:15.003Z mymachine.example.com
evntslog – ID47 [exampleSDID@0 iut=”3″ eventSource=
“Application” eventID=”1011″] BOMAn application
event log entry…
Rainer Gerhards [15:12]:
Rainer Gerhards [15:12]:
tell it all
Rainer Gerhards [15:12]:
so it could actually be
Rainer Gerhards [15:13]:
[winevent@ iut=”3″ eventSource=”Application” eventID=”1011″]
Rainer Gerhards [15:13]:
questions is .. do we need more than iut, eventSoruce and eventID to classify the message sufficiently?
Andre Lorbach [15:14]:
hrm well there is eventCategory, eventLogType and eventUser which migh tbe important
Rainer Gerhards [15:14]:
what was type again?
Andre Lorbach [15:14]:
EventLogType is “Application” or “System” for example
Rainer Gerhards [15:15]:
isn’t that source?
Andre Lorbach [15:15]:
source is the source machine
Andre Lorbach [15:15]:
or you mean eventsource ?
Rainer Gerhards [15:15]:
eventsource… yepp, that was what I wrote
Andre Lorbach [15:15]:
eventsource is “Adiscon WinSyslog” or “Adiscon Eventreporter”
Rainer Gerhards [15:16]:
doesn’t matter
Rainer Gerhards [15:16]:
I meant the log type “system”, “applicaton”, …
Andre Lorbach [15:16]:
hrm
Andre Lorbach [15:17]:
ok it could be eventSource=”ApplicationAdisconEventReporter”
Andre Lorbach [15:17]:
for example
Rainer Gerhards [15:17]:
let’s use the defined values http://www.mwagent.com/en/Manual/current/windowseventlogproperties.htm
Rainer Gerhards [15:17]:
I’d say we transfer in structured data:
Rainer Gerhards [15:18]:
id, user, netventlogtype
Andre Lorbach [15:18]:
but sourceproc shouldn’t be missing
Rainer Gerhards [15:18]:
and maybe severityid and, yesm, sourceproc
Rainer Gerhards [15:19]:
you are right, that should be required
Andre Lorbach [15:19]:
severityid is mapped to syslog priority
Rainer Gerhards [15:19]:
but its a lossy mapping
Andre Lorbach [15:19]:
ok thats a point
Rainer Gerhards [15:19]:
e.g. audit success becomes info
Rainer Gerhards [15:20]:
it would be best if the parser would understand a all properties
Rainer Gerhards [15:20]:
but accept if some were missing
Rainer Gerhards [15:20]:
too much work for an initial effort?
Andre Lorbach [15:20]:
it could be done easily with preg_match if we order the properties from mandetory to optional
Andre Lorbach [15:21]:
for example
Rainer Gerhards [15:21]:
that we can require
Andre Lorbach [15:21]:
id, user, sourceproc, netventlogtype is mandetory
Rainer Gerhards [15:21]:
lets keep stupid simple for the beginning
Rainer Gerhards [15:21]:
exactly
Andre Lorbach [15:21]:
but it can also be
id, user, sourceproc, netventlogtype, severityid
id, user, sourceproc, netventlogtype, severityid, category
id, user, sourceproc, netventlogtype, severityid, category, bdata
Rainer Gerhards [15:22]:
bdata is toooo much
Rainer Gerhards [15:22]:
I’d drop that idea
Andre Lorbach [15:22]:
ok leave bdata out
Rainer Gerhards [15:22]:
it scares me from the syslgod side
Andre Lorbach [15:22]:
then I would have 3 different preg_match calls for all
Rainer Gerhards [15:22]:
it won’t survive the 2k limit in any case
Rainer Gerhards [15:22]:
sounds good
Rainer Gerhards [15:22]:
so I’d say that’s decided
Andre Lorbach [15:22]:
ok I will note it down into my to do list … on emoment
Rainer Gerhards [15:23]:
we should also see that MonitorWare has a few default defintiions for this
Andre Lorbach [15:23]:
yes
Rainer Gerhards [15:23]:
would be useful and prevent users from getting frusrtrated – plus not a big deal to do
Andre Lorbach [15:23]:
three should be a button for it like for other products
Rainer Gerhards [15:23]:
exactly
Andre Lorbach [15:23]:
we made before
Rainer Gerhards [15:23]:
oh, one thing
Rainer Gerhards [15:23]:
if we go the -syslog-protocol path, we must be a bit careful
Rainer Gerhards [15:24]:
there are side-effects that could negativly affect us
Rainer Gerhards [15:24]:
I’d say we create the templates and see what happens
Andre Lorbach [15:24]:
kk
Rainer Gerhards [15:24]:
if it is too bad, we may use an interim template
Rainer Gerhards [15:24]:
e.g. replace [ by {
Rainer Gerhards [15:25]:
(the problem i see is that e.g. rsyslog *knows* syslog-protocol and does NOT put structured data into log files by default -these were not meant for humans)
Rainer Gerhards [15:25]:
*that* would be problematic
Andre Lorbach [15:25]:
k
Rainer Gerhards [15:25]:
of course, proper rsyslog configuration gets that, but another point that can get the user frustratec
Rainer Gerhards [15:26]:
ok, lets move on (I may need to be afk for a quick while but I let you know then)
Andre Lorbach [15:27]:
one minute …
Rainer Gerhards [15:28]:
ok, now I need a couple of minutes
Andre Lorbach [15:28]:
ok back
Andre Lorbach [15:28]:
lol ok …
Rainer Gerhards [15:28]:
will ping you
Andre Lorbach [15:29]:
k
Rainer Gerhards [15:42]:
back
Andre Lorbach [15:42]:
kk
Rainer Gerhards [15:43]:
so what’s the next point?
Andre Lorbach [15:43]:
ok lets begin from top of the list
Rainer Gerhards [15:43]:
ok
Andre Lorbach [15:43]:
– syslogtag, source filtered with OR like facility, priority
Andre Lorbach [15:43]:
this is how I implemented the filtering for now
Rainer Gerhards [15:44]:
I think that’s fine
Andre Lorbach [15:44]:
example
Andre Lorbach [15:44]:
ok
Rainer Gerhards [15:44]:
I got it
Andre Lorbach [15:44]:
because and filtering didnt make much sense
Rainer Gerhards [15:44]:
of course, it depends on user feedback
Rainer Gerhards [15:44]:
but it sounds intuitive and useful
Andre Lorbach [15:44]:
k
Andre Lorbach [15:44]:
– Add predefined searches into config? If yes how?
Andre Lorbach [15:44]:
next point
Rainer Gerhards [15:44]:
well
Andre Lorbach [15:44]:
we said we wanted to have a predefined selection of “searches” next to the search field
Rainer Gerhards [15:44]:
I’d say “yes”, but for the time being, these must be on a system-wide basis
Rainer Gerhards [15:45]:
because we do not yet have the user profiles
Andre Lorbach [15:45]:
ok … we wanted system wide predefined searches anyway
Rainer Gerhards [15:45]:
so I’d simply add them as config variables
Rainer Gerhards [15:45]:
lol
Rainer Gerhards [15:45]:
indeed
Andre Lorbach [15:45]:
so we just add user defined later
Rainer Gerhards [15:45]:
that I would suggest
Andre Lorbach [15:46]:
my idea to save them in the configuration is by kindly having a nother array, with two strings, one for the name of the search and one containing the filter string.
Rainer Gerhards [15:46]:
sounds perfect to me
Andre Lorbach [15:46]:
kk I will just note this down …
Andre Lorbach [15:47]:
– Making displayed columns configureable, how and which fields are available?
Rainer Gerhards [15:47]:
again… this time a system default
Rainer Gerhards [15:48]:
later taken from the user’s profile
Andre Lorbach [15:48]:
So we go for a fixed set of columns for now ?
Andre Lorbach [15:48]:
and make them configureable later ?
Rainer Gerhards [15:48]:
no, I’d go for configurable
Rainer Gerhards [15:48]:
but again, on a system-wide basis
Rainer Gerhards [15:48]:
(which also means the plumbing is there, but we “just” do not yet have the user-settings)
Andre Lorbach [15:50]:
ok … my best idea for now is also to use an array again, a single dimension array might be sufficient. each array entry contains the internal property name of the column
Rainer Gerhards [15:50]:
I concur on the array. I’d just use two values inside it
Rainer Gerhards [15:51]:
the name and also the POSITION
Rainer Gerhards [15:51]:
so where it is shown inside the table first column, second and so on
Rainer Gerhards [15:51]:
or do you take that simply from the array index?
Rainer Gerhards [15:51]:
lol… probably
Andre Lorbach [15:52]:
the name could be taken automatically by having the properties associated to names internally, and the position can be definied by simply moving the array entries up and down
Rainer Gerhards [15:52]:
so, yes just the name and it is ordered in order of appearance of the index (provided the index is monotonically incrementing)
Andre Lorbach [15:52]:
but this only works on a single dimensikon array
Andre Lorbach [15:52]:
sample
Rainer Gerhards [15:52]:
I got it
Andre Lorbach [15:52]:
$col[] = “prio”;
$col[] = “fac”;
…
$col[] = “msg”;
Andre Lorbach [15:52]:
you just move them as you need
Rainer Gerhards [15:52]:
sounds good
Andre Lorbach [15:54]:
next point: – split configuration files, or everything in one FILE?
Andre Lorbach [15:54]:
I would say for now one file, or ?
Rainer Gerhards [15:55]:
one file – and I honestly don’t see any use for multiple files in the future
Rainer Gerhards [15:55]:
(except for includes )
Rainer Gerhards [15:55]:
why would they be useful? What’s the idea behind it?
Andre Lorbach [15:56]:
no real idea, just to seperated db config from user config and so on… but I would also rather use one config file for now
Rainer Gerhards [15:56]:
let’s go for one and see if there actually ever pops up a real need to have multiple files
Andre Lorbach [15:56]:
kk
Andre Lorbach [15:56]:
next point: – Add Help Page
Rainer Gerhards [15:56]:
would be very nice to have
Andre Lorbach [15:57]:
maybe something for Florian later ?
Rainer Gerhards [15:57]:
but its pretty intuitive, so I don’t think its a showstopper
Rainer Gerhards [15:57]:
well… whoever
Rainer Gerhards [15:57]:
just lets keep the todo item
Andre Lorbach [15:57]:
k
Rainer Gerhards [15:57]:
(and just to make sure: we are talking abotu a one-pager, with maybe 200 lines or so?)
Andre Lorbach [15:58]:
I would go for one page, maybe with anchors it… like the SimpleMail manual, but with the full phplogcon framework of course
Rainer Gerhards [15:59]:
I concur – just wanted to evaluate required effort a bit
Rainer Gerhards [15:59]:
so its not a big deal in any case
Andre Lorbach [15:59]:
nope it isnÄ’t
Rainer Gerhards [15:59]:
I’d say we should bring it up once we know what exactly needs to go in
Andre Lorbach [15:59]:
ok
Andre Lorbach [16:00]:
then to the next point:
– What does “I’am feeling sad …” button?
Rainer Gerhards [16:01]:
hehe
Rainer Gerhards [16:01]:
I like that
Andre Lorbach [16:01]:
lol
Rainer Gerhards [16:01]:
first of all, would you object if we rename it to either “I’d like to feel sad” or “I am feeling too lucky”
Andre Lorbach [16:01]:
an image could show up, showing the linux pengiun shooting bill gates ?
Rainer Gerhards [16:01]:
nono
Andre Lorbach [16:01]:
lol
Rainer Gerhards [16:01]:
with the rename
Rainer Gerhards [16:02]:
I’d do a search for “error” inside msg
Rainer Gerhards [16:02]:
something that helps you feel sad ;=)
Andre Lorbach [16:02]:
lol
Rainer Gerhards [16:02]:
do we have a deal on that?
Andre Lorbach [16:02]:
Ok so the button actually would do a search for error messages
Andre Lorbach [16:02]:
sounds good to me xD
Rainer Gerhards [16:02]:
messages with “error” in them, to be precises
Rainer Gerhards [16:02]:
but the button must be renamed
Rainer Gerhards [16:03]:
and, to get serious again,
Rainer Gerhards [16:03]:
i’d do this via another set of config option: text for the button as well as query to execute
Rainer Gerhards [16:03]:
so one can acutally set it to something that is used really frequently
Rainer Gerhards [16:03]:
with the joke just being the default (of course )
Andre Lorbach [16:03]:
So it is a kind of a custom shortcut button
Andre Lorbach [16:03]:
hehe
Rainer Gerhards [16:03]:
jepp
Rainer Gerhards [16:04]:
I personally would find such a shortcut very useful
Andre Lorbach [16:04]:
i like r “I’d like to feel sad”
Rainer Gerhards [16:04]:
agree
Rainer Gerhards [16:04]:
heheh
Rainer Gerhards [16:04]:
or how about “make me feel sad”?
Andre Lorbach [16:05]:
lol ok
Rainer Gerhards [16:05]:
no, the other one’s better
Andre Lorbach [16:05]:
I’d like to feel sad ?
Rainer Gerhards [16:05]:
jepp
Andre Lorbach [16:05]:
k
Andre Lorbach [16:05]:
last discussion point:
– TimeFilter, continue reading logfile if timefilter failed, so if
Andre Lorbach [16:06]:
ok
Rainer Gerhards [16:06]:
mmmhhhh
Andre Lorbach [16:06]:
let me explain this further
Andre Lorbach [16:06]:
for example I want to view the events from last 24 hours.
Andre Lorbach [16:06]:
and we have the logfile stream
Andre Lorbach [16:07]:
the stream will be fully read if there are not more then 30 messages from the last 24 hours.
Andre Lorbach [16:07]:
Can we assume that a logfile will always be consistent in the date and time from beginning till the end ?
Rainer Gerhards [16:07]:
I’d say …. no
Rainer Gerhards [16:08]:
it very much depends on how the file is written
Andre Lorbach [16:08]:
only if we assume this, it could make sense to abort reading if time fails below 24 hours
Rainer Gerhards [16:08]:
by default, timestamps are taken from the message
Rainer Gerhards [16:08]:
as such, they can largely diverge
Andre Lorbach [16:08]:
otherwise we have to live with the overhead to read the whole log
Rainer Gerhards [16:08]:
maybe we should set a property in the stream’s config
Rainer Gerhards [16:08]:
because the overhead may be high
Rainer Gerhards [16:09]:
and with a proper template, I can ensure that the time log record is always advancing and consistent
Rainer Gerhards [16:09]:
so someone who tweaks the system could use that template
Andre Lorbach [16:09]:
I would say for the first beta we live with this overhead and add a stream config property later
Rainer Gerhards [16:09]:
and then define the stream to be “with consistent time”
Rainer Gerhards [16:09]:
makes sense
Rainer Gerhards [16:09]:
just add a TODO note… It’s easily forgotten
Andre Lorbach [16:10]:
I’ll put it on my TO DO after BETA ;)
Rainer Gerhards [16:10]:
ok
Rainer Gerhards [16:10]:
btw: isn’t that something that we could track as an enhancement request in bugzilla? or is it too early for that right now?
Andre Lorbach [16:11]:
I would start doing this after teh first beta
Rainer Gerhards [16:11]:
ok
Rainer Gerhards [16:11]:
whatever you like most
Rainer Gerhards [16:13]:
so we are done for now? :)
Andre Lorbach [16:13]:
ok this were all my discussions topics for now, then I have some todo’s before the BETA to reconcile with you
Rainer Gerhards [16:14]:
go ahea
Andre Lorbach [16:14]:
i just post them, if it is fine you just answer with yes or ok, an d I go on.
Andre Lorbach [16:14]:
– Add Basic configuration variables into config file.
Rainer Gerhards [16:14]:
I don’t fully get the point here
Andre Lorbach [16:14]:
ok this one is obvios
Andre Lorbach [16:14]:
lol
Rainer Gerhards [16:14]:
ok
Andre Lorbach [16:14]:
– Create installer script.
Rainer Gerhards [16:15]:
a lot of work?
Andre Lorbach [16:15]:
Guess not now
Andre Lorbach [16:15]:
I have a good sample of my installer from my stats project
Rainer Gerhards [16:15]:
ok, than its probably something to have starting from day one (its a must-have anyhow)
Andre Lorbach [16:15]:
I think i can easily adapt is
Rainer Gerhards [16:15]:
excellent :D
Andre Lorbach [16:15]:
+ for now we do not have to configure a database so its even more easy
Andre Lorbach [16:15]:
it will have to write a default config.php
Andre Lorbach [16:16]:
maybe we can decide what will be configured in the install procedure
Andre Lorbach [16:16]:
I would say, one default source for syslog messages. The default language
Rainer Gerhards [16:16]:
Well… I’d don’t try to do too much magic in the first placd
Rainer Gerhards [16:17]:
after all, folks need to know what they are doing
Rainer Gerhards [16:17]:
else it doesn’t play nicely with their syslog system
Andre Lorbach [16:17]:
I think the syslog source is mandeory
Rainer Gerhards [16:17]:
there are alos a number of security things…
Andre Lorbach [16:18]:
a big fat warning will be shown of course if the install script is still available after installation
Rainer Gerhards [16:18]:
well… what I want to convey… I think those that will try the intial version will be able to change the settings inside the config file as long as they are sufficiently well pointd at it
Andre Lorbach [16:18]:
yes sure
Andre Lorbach [16:18]:
it just will help to easily create a default config
Andre Lorbach [16:19]:
let me show you how the stats installer looks like
Rainer Gerhards [16:19]:
ok…
….
Andre Lorbach [16:20]:
you will see the screenshots in this thread
Rainer Gerhards [16:20]:
already browsing…
Rainer Gerhards [16:21]:
looks very good
Rainer Gerhards [16:21]:
and already proven in practice
Rainer Gerhards [16:21]:
I like that ;)
Andre Lorbach [16:21]:
hehe
Andre Lorbach [16:21]:
the install system is similar to the postnuke one
Rainer Gerhards [16:22]:
so its also along the line of what people expect?
Rainer Gerhards [16:22]:
that would be even better…
Andre Lorbach [16:22]:
its a common step by step install
Rainer Gerhards [16:22]:
excellent
Rainer Gerhards [16:23]:
so its agreed upon
Andre Lorbach [16:25]:
very well
Andre Lorbach [16:26]:
– Implement DB Driver
Rainer Gerhards [16:26]:
ohhhhh… yes
Rainer Gerhards [16:26]:
last thing before initial release
Andre Lorbach [16:26]:
k
Rainer Gerhards [16:26]:
and maybe first thing after it
Rainer Gerhards [16:26]:
I’d say….
Andre Lorbach [16:26]:
– Fix Pager
thats more a note for me
Rainer Gerhards [16:26]:
let me quickly go back
Andre Lorbach [16:26]:
k
Rainer Gerhards [16:26]:
as I already said, I’d expect that the db driver is quite trivial
Rainer Gerhards [16:27]:
I’d *NOT* go for optimization in the first step
Andre Lorbach [16:27]:
ok
Rainer Gerhards [16:27]:
so in essence it is just pulling the records from db, sorted on the uid and a bit glue to not pull a full result set every time
Rainer Gerhards [16:27]:
do you think it is much bigger?
Andre Lorbach [16:28]:
depends on how dynamic it has to be … but I think it will become clear when I just start doing the DB Driver
Rainer Gerhards [16:28]:
yeah.. that’s right
Rainer Gerhards [16:28]:
lets postpone the discussion
Andre Lorbach [16:28]:
only minor points left
Andre Lorbach [16:29]:
– Add Text Highlight Feature
Andre Lorbach [16:29]:
and
– Show Details + Full Message on MouseOver on Message field
Andre Lorbach [16:29]:
like in php-syslogng
Rainer Gerhards [16:29]:
that sounds like a good feature
Rainer Gerhards [16:29]:
ah…
Rainer Gerhards [16:29]:
one point
Rainer Gerhards [16:29]:
I think we do not currently have that
Rainer Gerhards [16:29]:
we have the list view
Rainer Gerhards [16:29]:
did we already discuss detail view?
Rainer Gerhards [16:30]:
that is what should be displayed when somebody clicks on a row in list view
Andre Lorbach [16:30]:
we have none discussed yet
Rainer Gerhards [16:30]:
much like in phpLogCon v1
Andre Lorbach [16:30]:
ok
Rainer Gerhards [16:30]:
IMHO its not necessarily something need to have for intial release
Rainer Gerhards [16:30]:
but shortly after that
Andre Lorbach [16:30]:
ok
Rainer Gerhards [16:31]:
a biggie?
Andre Lorbach [16:31]:
no
Andre Lorbach [16:31]:
don’t think so
Rainer Gerhards [16:31]:
puuuh… ;)
Andre Lorbach [16:32]:
ok that is all on my list
Rainer Gerhards [16:32]:
one thought on the full message
Andre Lorbach [16:32]:
i will send you what I have written down for verification
Rainer Gerhards [16:32]:
ok
Rainer Gerhards [16:32]:
but…
Rainer Gerhards [16:32]:
I just thought that this may be quite a lot of data
Rainer Gerhards [16:32]:
in the long term, there should be an option to restrict the displayed size (first n chars) or turn it off altogether
Rainer Gerhards [16:33]:
I think that woud make sense
Andre Lorbach [16:33]:
you mean in the list view ?
Rainer Gerhards [16:33]:
the popup
Andre Lorbach [16:33]:
ah ok
Rainer Gerhards [16:33]:
but also in listview, you are right
Rainer Gerhards [16:33]:
think of an nt even message
Andre Lorbach [16:33]:
its done in the listview already#
Rainer Gerhards [16:33]:
2k in size…
Rainer Gerhards [16:33]:
:)
Andre Lorbach [16:33]:
but it should be configurable
Andre Lorbach [16:33]:
the char limit
Rainer Gerhards [16:33]:
by all means
Rainer Gerhards [16:34]:
and while we are at it, we may also add an option to turn the popup off
Rainer Gerhards [16:34]:
that’s not much work ;)
Andre Lorbach [16:34]:
ok
Rainer Gerhards [16:34]:
once again: do you have any concerns if I post the chatlog for others to see?
Andre Lorbach [16:35]:
no problem