FreshBSD

    that log_debug is no longer needed

    in session_read_data() do NOT enable back read event otherwise it will
    corrupt the session and cause a fatal.
    
    reported by Roman Kravchuk, with this diff in OpenSMTPD survives again the
    evil smtp_tester.py :-p

    Do not call session_respond() AND session_imsg() from the same handler as
    they are exclusive. In the quit handler, it will lead to a corrupted
    session as the F_QUIT is set and session_respond() will kill the session
    before session_imsg() is done.
    
    For now, comment session_imsg(IMSG_MFA_QUIT) as we don't have filters
    enabled anyways. The proper fix will be to have session_imsg() called
    from the handler THEN upon return in smtp process do the session_respond()
    call.
    
    Fixes a segv reported by Roman Kravchuk. While at it, revert my last commit
    in session_error() which was probably needed because of this bug.

    in session_error(), do not call session_destroy() when the F_WRITEONLY
    session flag is set, set F_QUIT instead.
    
    fixes a "corrupted session" fatalx() that triggers when a client
    disconnects after sending the end of message '.' but without waiting
    for server to acknowledge.
    
    reported by Roman Kravchuk <kravchuk.kp at gmail.com> a while ago, could not
    reproduce until today.

    we should never use ramqueue_first_envelope() as a mean to determine the
    first schedulable envelope otherwise we end up with nsched wrong ...
    
    heck, let's kill ramqueue_first_envelope() and be done with it ;)

    the ramqueue filling at startup was busted, it would load envelopes one by
    one instead of doing it efficiently.
    
    fix runner_timeout() and ramqueue_load() so that at startup smtpd fills the
    ramqueue as long as there's no schedulable envelope in it, interrupts
    filling if there is and resume once it's scheduled.
    
    bug spotted by Nathanael Rensel, bug fix by me w/ help from eric@
    tested by eric@ and I

    now that we no longer do a stateful iteration on schedule queue, we need
    to make ramqueue_next_envelope() aware of pauses in mda/mta. while at it
    kill the pausing of bounces since they are reinjected in smtp and end up
    paused by the mda/mta cases.
    
    fixes an infinite loop observed by eric@ when pausing deliveries and
    trying to interrupt smtpd while it attempts to fetch next envelope :)
    
    tested by eric@ and I, ok eric@

    remove stateful iteration from ramqueue, if we ever need to reintroduce it
    we'll do it, but it isn't used and causes potential bugs
    
    idea by Nathanael Rensel, diff by me, ok eric@

    in ramqueue_schedule(), do not reorder ramqueue envelopes when forcing a
    reschedule of the entire queue as that can lead to an infinite loop.
    
    bug spotted and bugfix by Nathanael Rensen, thanks !
    
    ok eric@

    queue_fsqueue_ascii.c, you're not welcome anymore

    implement an envelope_ascii API that's not tied to a specific queue_backend
    simplify queue_fsqueue

    enable back CA support, just don't verify client ...
    
    tested @ home

    *finally* make use of certificate authority file if available !
    
    bits from relayd, ok chl@, ok eric@

    - man page had an example wrong
    
    spotted and diff by Ptter J. Philipp <php at centroid.eu> a while ago, thanks

    - introduce delivery backend API (delivery.c)
    - move each delivery method to it's own delivery backend
    - simplify smtpd.c accordingly
    - rename A_EXT -> A_MDA since that's what we really do
    
    ok eric@

    - change callback prototypes in filter.c to allow stricter checks
    - introduce STATUS_WAITING, filters will reroute async DNS queries through
      LKA in a near future
    - filter_api.c will contain our filter API (empty for now)
    - Makefile to build libsmtpdfilter (not linked to the build,dev stuff only)

    wooooops defines missing from previous commit

    Qwalk, our API to linearly walk over the persistent queue, did not take the
    queue_backend into account and assumed a filesystem with a specific layout.
    
    This commit does plenty of things:
    
    - make qwalk an abstraction in the queue_backend API, and impose queue
      drivers to implement qwalk_open(), qwalk() and qwalk_close();
    
    - move previous qwalk_open(), qwalk() and qwalk_close() to the fsqueue
      driver since they were fsqueue specific ...
    
    - make qwalk API work with msgid/evpid instead of pathnames since we're
      going to use the queue_backend API to load envelopes by evpid anyway;
    
    - makes smtpd use *solely* the queue_backend API when manipulating the
      queue. pathnames were removed from smtpd.h and moved into the fsqueue
      which means we can now store a queue anywhere ... as long as we write
      the ten functions or so required for a queue driver ;-)
    
    ok eric@, ok chl@

    - fix smtpctl pause/resume so the ramqueue scheduling is done correctly
    - rename IMSG and smtpctl pause/resume parameters
    - update man page
    
    tested by me, ok chl@, eric@

    - smtpctl schedule no longer works, instead, use 'smtpctl schedule-id <id>'
    - introduce 'smtpctl schedule-all'
    
    ok eric@

    introduce Q_CORRUPT and queue_backend operation to move a message from
    schedule queue to corrupt queue upon envelope loading failure.
    
    tested by me, ok eric@

    fsqueue no longer stores envelopes by dumping the structure, instead use a
    couple of load/dump functions to convert to and from a human readable fmt.
    while at it kill struct delivery and merge back its fields to the envelope.
    
    this basically means we shouldn't require users to flush their queues every
    time we make a change to struct envelope.
    
    work is not done, but we're at a better state than the binary fsqueue so
    we'll improve it in-tree.
    
    has been running on my own box for the last 12 hours or so
    ok eric@, chl@

    fix parsing of :include: aliases

    document that virtual maps require a domain key

    - fix a segv caused by a deep recursion introduced with a recent quick-fix
      and triggering if clients sent a large number of DATA lines in one write

    clarify that network sources are to be specified in CIDR notation
    
    by David Walker, ok jmc@

    removing annoying debug log

    a single ramqueue message may be shared by many ramqueue envelopes to be
    delivered to many ramqueue hosts, therefore storing the rq_host pointer in
    the rq_msg envelope is wrong and causes baaaaad behavior.
    
    this commit fixes reliability issues in runner process, experienced and
    reported by many

    - introduce filtermask in struct smtpd
    - do not forward lines to mfa when FILTER_DATALINE is not set in filtermask
    
    prevents smtpd from handling mails slowly while I'm hacking on filters
    support

    move a log_debug() call to a less annoying spot so that it does not get
    called for each single line of DATA exchanged between smtp and mfa

    add support for per-line DATA callbacks, this allows filters to take their
    decisions *while* the message is being received by the client.

    fix pasto in strlcpy() truncation test

    typo

    initial support for a session-time filtering API
    
    currently only HELO/EHLO, MAIL, RCPT are supported, however ... I have
    voluntarily disabled filters at smtpd.conf level so people don't play with
    it until the API has stabilized a bit
    
    discussed with several people in private, no one opposed the feature

    - plug a memory leak in runner_remove_envelope()
    - cosmetic change

    move ramqueue_host pointer from ramqueue_envelope to ramqueue_message.
    this allows us to save one pointer from each envelope stored in ram while
    still allowing O(1) host lookups by ramqueue_envelope.

    - teach smtpctl remove about the new ramqueue structure
    - bonus #1: O(log n) removal of envelopes
    - bonus #2: removing all envelopes that have the same msgid works again

    - introduce ramqueue_lookup_{host,message,envelope} to perform lookups in
      the new ramqueue structure
    - introduce ramqueue_reschedule() and ramqueue_reschedule_envelope() which
      to reschedule a message or a specific envelope.
    
    O(n) ->  O(log n)  \o/

    smtpctl show sizes, displays the size of queue-related structures, useful
    for developers to see the impact of structure changes on memory and disk
    usage, and useful for users to better understand 'smtpctl show stats'

    add a host-tree and an envelope-tree in the ramqueue, they will be used to
    improve scheduling and general ramqueue operations. unused yet

    - update smtpctl.8 to reflect reality
    - bring back 'smtpctl schedule' and 'smtpctl remove' to life
    
    Things you should know:
    
    The ramqueue data structure is not finished yet and lacks an envelope tree
    for evpid lookups. I wanted to wait until I'm done but too many people are
    affected by not being able to reschedule envelopes, this is a quick fix.
    
    So right now there's an O(rrible) complexity as both commands will perform
    a (possibly aborted) queue scan leading to O(n). I will make that O(log n)
    soon.
    
    Also, smtpctl remove no longer supports removing an entire message, I will
    fix that very soon too.

    fix that fscking "fgetln(): bad file descriptor" bug experienced by many.
    
    long story short: datafp is reused by the possibly multiple client sessions
    of a MTA session. fclosing it in client_close() will cause the file pointer
    to be invalidated as soon as we fail the primary MX and attempt the second.
    
    bug introduced while trying to fix a leak causing a crash for a user, now
    smtpd will unconditionnally fclose() in MTA_DONE and if a leak still exists
    then we should track why the mta session doesnt reach MTA_DONE.

    %u in format string is rcpt.user not rcpt.domain
    
    From Tim van der Molen <tbvdm at xs4all.nl>

    'relay as' and 'relay via as' rules allow smtpd to rewrite the user part,
    the domain part or the entire address of the sender at the SMTP sesssion
    level. this is not masquerade but allows smtpd to communicate with hosts
    that do a check of SMTP sender fqdn.
    
    sent to tech@, a couple 'no regression' feedbacks

    teach smtpd how to listen on an interface group so that we can do:
            listen on egress
            listen on wlan
    
    idea unvoluntarily suggested by Mikolaj Kucharski a few weeks ago,
    unslacked after theo suggested it again.

    fix examples so they stay do not use external utilities and do not refer
    to external domains
    
    prompted by deraadt@

    remove unused commented structure

    until now the map_backend API was not really useful for backends that are
    not key/val stores. refactored a bit so that smtpd can really take
    advantage of backends. preliminary work for ldap support ;-)
    
    no functionnal change

    remove annoying log_debug()

    when a user add himself to his own ~/.forward, the delivery method was not
    reset by ruleset matching and since it was cleared upon entering
    lka_session_resolve_node(), it would trigger a fatal().