MM edit history - Frank da Cruz Beginning with MM-0.90.4 (Notes are chronological, newest stuff at the bottom). Warning: Includes learning experiences and false starts. ------------------------------ Wed Jan 9 19:42:31 2002 Short story: I added some code to MM to address the truncating problem that occurs when you edit an outgoing message, save the result, and exit from the editor. Long version: I spoke with Melissa about MM and found out where the latest (patched) sources were: /src/acis/ccmd/src /src/acis/mm/obj/solaris25 I made an ~fdc/mm/ directory on prost with mm and ccmd subdirectories. Copied all the files from the /src tree. Checked that they built OK; in ~/mm/ccmd: make solaris In ~/mm/mm (it looks to me like the current Makefile unconditionally defines SOLARIS?): make This makes an MM 0.90.4(xx) binary: Columbia MM, version 0.90.4(24) Copyright (c) 1986, 1990 The Trustees of Columbia University in the City of New York Compiled by fdc@watsol on 9 Jan 2002 6:16pm Report bugs to bug-mm Now the truncating problem: The editor is run by mm_execute() in exit.c, which is just a fork() and execvp(). Then the upper fork calls wait_for_process(pid), which calls do_wait(pid, true), where "true" means "blocking". do_wait() might need looking at but first assume it works as advertised. Where is Howie's sleep()? Nowhere. I added an msleep() (millisecond sleep) routine to datime.c, for Solaris only. Uses select(). Builds without complaint. Then I added a chk_edit(filename) routine to edit.c, which sleeps for half a second and then in a loop, gets the size of the filename, compares it with the previous size, and if they are not equal, sleeps for 1/10 second and tries again, up to 25 times. If, after 25 iterations, the size still doesn't settle down, or some other bad thing happens, chk_edit() returns < 1. Then I found all the places in edit.c where mm_execute() is called with editor[0], and after each of these, I called chk_edit() on the temp file. If chk_edit() returns < 1, we set a trouble flag, which is supposed to inhibit deletion of the temp file. The very first time I tried it, I caught it in the act: (Exit from EMACS here) /p/kd/fdc/.mm-outgoing.25233: 0 6263 (BAD) /p/kd/fdc/.mm-outgoing.25233: 6263 6263 (OK) S> This seems to exonerate wait_for_process() (e.g. of killing EMACS prematurely). The messages won't appear unless there is at least one size mismatch. If the size doesn't settle down after 3 seconds (500 msec + 25 x 100 msec), it prints "Possible problem with edit, temp file saved: " and the filename. I didn't notice any kind of edit history file, but a copy of this message is in ~fdc/mm/NOTES.fdc. I'll be the guinea pig, feel free to join in; time will tell. - Frank P.S. The fact that the BUG command puts "SunOS" in the Bug-Report: header doesn't really mean SunOS -- the Solaris versions really are compiled for Solaris, but the real name of Solaris is SunOS. P.P.S. This message itself was truncated when I exited from EMACS, but the new code fixed it automatically. P.P.P.S. Melissa or Postmaster, I guess you can add me to the bug-mm list :-) ------------------------------ Date: Thu, 10 Jan 2002 9:40:01 EST From: Melissa Metz To: Frank da Cruz Cc: bug-mm@columbia.edu, kermites@columbia.edu, unixsys@columbia.edu, postmaster@columbia.edu, Vace Kundakci , Walter Bourne Subject: Re: MM truncating problem maybe fixed? In-Reply-To: Your message of Wed, 9 Jan 2002 20:20:27 EST Message-ID: Cool. > I didn't notice any kind of edit history file, but a copy of this > message is in ~/mm/NOTES.fdc. We kept track of the edit histories in RCS. e.g. cd /src/acis/mm/src, co -l edit.c, , ci -u datime.c, . The comment can be as long as you like; you can use "ci -u edit.c P.S. The fact that the BUG command puts "SunOS" in the Bug-Report: > header doesn't really mean SunOS -- the Solaris versions really are > compiled for Solaris, but the real name of Solaris is SunOS. Right, uname also reflects the name SunOS. Sun claims that Solaris is the whole operating system and all applications, while SunOS is just the kernel. --Melissa ------------------------------ Thu Jan 10 09:24:55 2002 Various refinements and tuning, #ifdef adjustment, message improvement, etc. The main impact of these changes is (a) there is always at least a 500-msec delay after returning from EMACS, and (b) if you return from EMACS with an empty file, there's a longer delay because MM has no way of knowing if the file is empty because it hasn't been written yet (this has happened to me many times), or because you really saved an empty file. read_from_temp(), which is called after every editor invocation, executes an fsync() on the file to forestall truncation, but doesn't. I suspect fsync() has no effect on a file open with O_RDONLY. The Solaris 2.5.1 man page suggests as much: ERRORS EBADF: fildes is not a valid file descriptor open for writing. although in Solaris 2.5.1 fsync() on a read-only file returns 0 and does not set errno. The Solaris 8 man page, on the other hand, says "EBADF: The fildes argument is not a valid file descriptor". I wonder if EMACS does an fsync(), and if it does, how can MM be reading partial files? ------------------------------ Sat Jan 12 15:25:07 2002 At some point in the last few years, MM started adding a blank line to the end of every message after returning from the editor. Off-by-one bug at the end of read_from_temp(), fixed in edit.c. Changed copyright date from "1986, 1990" to "1986, 2002". This makes it a bit more obvious that it is being maintained, handles Y2K, etc. I made this change only in misc.c to keep the differences small. ------------------------------ Sun Jan 13 11:34:11 2002 Changed #ifdefs around new code from #ifdef SOLARIS to #if defined(BSD) || defined(_AIX) || defined(SVR4), since (a) this is what is used for dttzone(), which depends on the same struct definitions (timeval and timezone) and functions (gettimeofday()), and (b) to include SunOS, which also has/had the truncating problem. datime.c, edit.c. But surprisingly, this did not pick up Solaris -- I assumed that defining SOLARIS would imply SVR4 but it doesn't, so I changed the #ifdefs for the new code to: #if defined(BSD) || defined(SOLARIS) || defined(SVR4) || defined(_AIX) But I did not change the #ifdefs around dttzone(). datime.c, edit.c. Tried building for SunOS: ccmd: "make clean; make sun" failed at link time, but then worked if I manually deleted ccmd.a, test.o, and skel.o and did "make sun" again. mm: "make clean; make" fails because of the hardwired SOLARIS definition in Makefile. Copied the Makefile out of /src/acis/mm/obj/sunos41 and used that instead and mm build ok. Tried building for Solaris 8 on newcunix in /p/kd/fdc/mm/solaris8/: ccmd: "make clean; rm -f ccmd.a test.o skel.o; make solaris" (ok) mm: "make clean; make" (ok) Tried building for HP-UX 11.20 but it bombed right away with undefined symbols. Didn't try fixing or building on AIX or anywhere else. In heavy testing and real use of MM over the past several days, I have not experienced a single truncation (normally I have 3-4 per day). However, after rearranging the code on Thursday, I also have not seen a single instance of the file size changing. The 500-msec pause appears to be sufficient, but the subsequent size-checking loop is still there as a precaution (e.g. for high load, memory full, swap-space low). I'd suggest the MM version number be bumped up and the new source replace the old source on the public FTP server. ------------------------------ Other things on my list: Maybe add a SET command to enable/disable chk_edit() since most people don't need it. Check and enable or fill in msleep() for other platforms. Character sets, when sending a message [done, see below]: . Don't strip 8th bit from text (what is 8th bit used for now?) . SET CHARACTER-SET (default US-ASCII), SHOW CHARACTER-SET . Add header: Content-Type: text/plain; charset="iso-8859-1" or whatever. . Add header: Content-Transfer-Encoding: 7bit (or 8bit) according to charset. . Add header: MIME-version <-- get right syntax. MM shouldn't "do" anything with character sets, just let the user declare which one is used in the outbound message and not strip the 8th bit from text, Subject, or other header data characters. The SET CHARACTER-SET command would take a keyword list of MIME charset names for only those MIME charsets that are also international standard sets (i.e. ISO registered), since it's impolite and antisocial to send email in proprietary character sets; an internal table tells whether each one is 7bit or 8bit. The commands and headers are easy, but figuring how to undo whatever is currently done with the 8th bit is hard. Slightly more ambitious tasks: . Add switches (cmswi()) to some of the commands; for example: COPY /APPEND /REPLACE /BODY-ONLY . COPY and MOVE to pipelines. . Hooks to Kermit, e.g. COPY /DOWNLOAD [done, see below] . Allow definition of macros . Allow macros to accept parameters . Case-sensitive text searches. . Pattern or regexp text searches. . Devise a way of accomplishing OR in message selection; currently conditions are all connected by implied ANDs. This would also require syntax for grouping. . Add an option that tells MM, when it reads a message and finds a charset header, to send the corresponding ISO 2022 escape sequence to switch the terminal into the character set used by the message or, if there is no charset header, to switch it back into its prevailing charset (such as Latin-1). Of course this can work only with standard ISO-registered character sets, not with (e.g.) Windows code pages or random MIME charsets, and only with ISO-2022 / ANSI X3.64 compliant terminals (e.g. VT220 or 320). The latter would require MM to have to several distinct charset settings: . The default (global) charset for sending . The default charset for reading (when message doesn't specify one) . A way to send a message with a given charset without affecting the global one (e.g. SEND /CHARACTER-SET:blah). . A way to tell MM, when replying, whether to automatically choose the same charset as the original message (again, without affecting the global outbound charset), e.g. REPLY /SAME-CHARSET or something, plus a global setting for whether this should be the default. Also: . Consolidate the many makefiles and make a master makefile that builds both ccmd and mm [done, see below]. . Clean up compiler warnings [mostly done]. . Check/update/fix ports to HP-UX, AIX, FreeBSD, etc. Definitely NOT on my list: . Enclosures (if I want this I'll use another mail client!) . Desktop metaphors (folders, in-boxes, address books, "My New Messages", "My Favorite Friends", etc -- fooey, ditto!) ------------------------------ Wed Jan 16 13:12:29 2002 A message was lost -- first time in nearly week. I was replying to a message and editing the reply. I was typing really fast, including the ^X^S^X^C to save and exit EMACS. In this case the MM S> prompt appeared immediately, with no 0.5sec pause. Therefore there must be a path thru the code in which chk_edit() is not called. (Later the same day several more messages were truncated; same deal: no pause.) ------------------------------ Wed Jan 23 14:02:56 2002 A reply was totally lost -- 0 bytes -- but this time MM paused for the full 3 seconds. ------------------------------ Sun Feb 3 11:20:12 2002 IDEA: add SET TMPDIR to allow temp files to be written on a local disk. e.g. "set tmpdir /tmp/fdc". Create it if it doesn't exist. Give it 600 perms. But what if it does exist and is not owned by the MM user? Give an error message. Give all temp files mode 600 on open. ------------------------------ Mon Feb 4 16:53:13 2002 Many composed messages & replies lost. A big problem is that MM always deletes the .mm-xxx tmp file after reading it in (unless it *knows* there's a problem). Not deleting it would at least let us get it back but then: (a) successive messages use the same filename (.mm-outgoing-), and (b) when *would* we delete them? <-- When MM EXITs. ------------------------------ Mon Apr 15 16:20:53 2002 Built Ken Harrenstein's updated version (see KLH.NOTES) and started running it in debug mode ("make fdc-solaris"). It built fine on Solaris, works fine so far. When exiting from EMACS, we always get the debug message: "wait3 errno 2" (no such file or directory). Looked through sources again for any invocations of the editor that do not call chk_edit() afterwards; didn't find any. Added more debugging statements. ------------------------------ Tue May 14 11:54:37 2002 Melissa was getting more truncations. Replaced the version of read_from_temp() that does calloc() and then (potentially) a bunch of rmallocs() with the original simple version that does just one malloc(), and added debugging after every operation. Suspicion: read() and malloc() take an int, but the old code was calling it with sbuf.st_size, which is long. Changed the code to (a) convert the long to an int, (b) check that (long) of the int is equal to the original long, and then (c) to use the int in all the malloc() and read() calls. ------------------------------ Thu May 16 19:20:52 2002 Melissa was still getting truncations. The messages indicated that everything in read_from_temp() was working right. But upon return the message is empty. That means that the pointer to the message is not being returned right. Aha. The last statement in read_from_temp() is return(newtext). But newtext is on the stack; made it static (it shouldn't matter but you never know). Added debugging checks and messages to read_from_temp()'s callers. ------------------------------ Tue May 21 17:44:21 2002 Replying to message, entered EMACS, edited the reply, exited EMACS, debugging messages showed everything normal, yet the message was truncated to 0 bytes. There should be NO path through the code where that can happen without a message. At this point we still have two problem cases: 1. Debugging messages all come out normal, but file is stil truncated. 2. There are no debugging messages at all, and file is truncated. ------------------------ Sun Jul 21 15:15:47 2002 Fixed stupid debugging message for empty temp file: Text length mismatch [gnuemacs_edit_outgoing]: 0 != 0. Added more debugging messages, including a message when a missing final terminator is supplied in case this correlates with truncation. edit.c. Added "set user-temp-files-keep" (keyword chosen so it goes at end of list, since it wasn't clear to me how to insert a new keyword -- do I have to change all the #defines in set.h?). Made the default for this "yes" if FDC_EDITFIX is defined, since otherwise what good is it? set.h, set.c, extern.h. Filled in action for "set user-temp-files-keep yes" -- just check the boolean variable for this setting in purge() and change all open()s for temporary files to include O_TRUNC (since we keep recycling the same name). Then in cmd_quit(), delete the temp files (i.e. when user says BYE or QUIT). Thus after editing a message, the temp files stay around until the next time you enter the editor from within MM, or you (really) exit from MM, so now if any message is truncated after editing, you can get it back from the temp file -- and if the temp file itself is truncated, we know it's the editor's fault and not MM's. edit.c. ------------------------ Fri Sep 13 16:06:03 2002 Enclosed all verbose debugging messages in edit.c in #ifdef VERBOSE_MESSAGES except "Temp file kept: " in gnuemacs_edit_outgoing(). ------------------------ Sat Sep 14 13:31:27 2002 The 8th bit of data characters seems to be used by CCMD to mark whether each character was (or is to be) echoed or quoted or something; see CC_CHR, CC_QUO, etc, in ccmd.h and mm/parse.c. But this stuff should apply to commands, not data text. 1. Send, ^E, create 8-bit text in EMACS, exit from EMACS, send: Stripped. 2. Send, ^B, insert 8-bit file, send immediately: Stripped. 3. Send, type 8-bit chars directly into text buffer: Stripped. In case 3, it's obviously because the text is being read by CCMD. But in cases 1 and 2, there must be some code that zips through the new text and strips off the bits. When EMACS returns (and also when "Insert file:" inserts data from a file), the data is still 8-bit, so the bits are being stripped later. In cmd_send() in sendcmds.c, upon return from get_msg(&outgoing), the 8th bits are gone. OK, so get_msg() sets up Ctrl-E handler (editaction()) and then calls parse(), which calls the editor if I type Ctrl-E, but then strips the bits. BUT... if you "set use-editor-always", parse is not called, and the bits are not stripped. This works even when replying to messages. ------------------------ Sun Sep 15 14:19:48 2002 Adding new header items... The idea is that normally these would be omitted, but if the user gives a CONTENT-TYPE command, the other two would be filled in too, using default values. Use "reply-to" as a model. Commands for the others would be available to override the default values. message.h: [ok] Add symbols for new headers: #define MIME_VERSION 21 #define CONTENT_TYPE 22 #define TRANSFER_ENCODING 23 [ok] Add new header members to struct mail_msg definition. parsemsg.c: [ok] Add header keywords to header_keys[] -> header_tab -> header_fdb... [xx] Add cases to parse_msg()... (not necessary). cmds.H: [ok] Add CMD_ symbols for CONTENT-TYPE, MIME-VERSION, and TRANSFER-ENCODING. cmds.c: [ok] Add CONTENT-TYPE, MIME-VERSION, and TRANSFER-ENCODING to send_keys2[]. [ok] Add them to hdr_keys[] too. [ok] Add prototypes. [ok] Add to dispatch table mm_cmds[]. display.c: [ok] Add cases to cmd_display(). send.c: [ok] Add action routines: cmd_content_type(), etc. NOTE: "help header-fields" is missing. ------------------------ Mon Sep 16 10:14:55 2002 Updated copyright date in VERSION text, misc.c. ------------------------ Tue Sep 17 10:11:38 2002 Fixed Linux build with help from and ... ccmd/Makefile test and skel targets should not have hardwired -ltermlib. Added TERMLIB symbol to ccmd Makefile to handle this. ccmd/machdep.h #define HAVE_BSTRING and HAVE_TZ_SYSV for Linux. mm/Makefile In Linux, link with ncurses, not termlib; handled in Linux target. mm/sysh/s-linux.h define NEED_TIME and HAVE_BSTRING mm/config.h Contains hardwired #include "sysh/s-sun58.h". Noticed that there are a bunch of config.h-blah files, where blah == platform-name, and each one #includes a different "sys/s-blah.h". Made one for Linux (and while I was at it, for HP-UX, AIX, etc, but these are probably not right). Then I changed each target in the MM makefile to link config.h to the appropriate config.h-blah. We still get warnings about mktemp() but so what; unfortunately mkstemp() is not a compatible replacement, so if somebody really cares about this, they'll have to write an mm_mktemp() routine that handles both cases. Anyway, once I installed movemail in /usr/local/lib/mm/ Linux MM works fine. ------------------------ Tue Sep 17 20:04:29 2002 Adding a new command... DOWNLOAD command for Read Mode, plus SET/SHOW DOWNLOAD-FILTER. Works just like PRINT and PRINT-FILTER but downloads rather than prints. cmds.H: [ok] Add CMD_blah symbol (in alphabetical order). cmds.c: [ok] Add DOWNLOAD keyword to mm_top_keys_2[]. [ok] Add DOWNLOAD keyword to mm_send_keys_inv[]. [ok] Add DOWNLOAD keyword to mm_read_keys_2[]. [ok] Add prototype for cmd_download(). [ok] Add cmd_download function pointer to mm_cmds[]. print.c: [ok] Copy cmd_print() to cmd_download() with minor edits. [ok] Modify cmd_download() to have a /FILENAME: switch. [ok] Modify cmd_literal() to allow LITERAL DOWNLOAD. set.H: [ok] Define SET_DOWNLOAD_FILTER set.c: [ok] Define default download_filter = "kermit -s - -a %s". [ok] Add "download-filter" to set_cmds. [ok] Add to download_filter to set_variables[]. extern.h: [ok] Add download_filter to list of extern strings. Combined notes on new features into release-0.91.txt. Moved old makefiles into mm/old/ and ccmd/old/. Updated help files. ------------------------ Wed Sep 18 12:39:40 2002 Changed default download filter to "kermit -Ts -s - -a %s" (added the 'T') to ensure that Kermit downloads in text mode. set.c. Changed DOWNLOAD command to allow up to four %s's in the download-filter (and to fill them in without overflowing any buffers) for the benefit of setups like "set download-filter cat > %s; sz %s; rm %s". print.c. Figured out how to set the breakmask for the _CMFLD call in cmd_download() to allow all printable characters (except ?), so now you can type DOWNLOAD /FILE:c:\foo$bar_baz\xxx.eml or /FILE:~/mail/xxx.tml. print.c. I noticed an apparent problem with the HEADERS command -- dates were coming out wrong. I thought I had broken something, even though I never went near any code that deals with dates. It turns out to be a longstanding problem. Quoting Melissa: "This is the creeping-DST bug. It hasn't always been there. We've fixed it many times. But DST is pretty slippery. MM misinterprets the DST status of the message, so each time it writes out the file, you quit and restart MM, then read the file again, the time creeps +/- 1 hour. Maybe a difference in the DST or time() library or timezone setting or something? "If you are restarting MM a lot (e.g. debugging), the message creeps 1 hour each time, and after 31 iterations 9/17 16:00 (GMT) becomes 9/16 09:00. You can [see the effect] by copying a file, reading it in, writing it out again, and running diff to see if the time changed. "You could 'normalize' the envelope times by moving all the messages into a no-envelope format like RMAIL, then reading it back in with MM (which will initialize using the Date: header) and writing it back out in your desired format." Envelope format refers to mbox-style message files in which each message starts with a "From " line. MM uses this line, rather than the Date: header, to determine the message's timestamp. The bug is the MM recalculates the timestamp incorrectly and rewrites the "From " line every time it writes out the mail file. ------------------------ Thu Sep 19 10:56:36 2002 When a CONTENT-TYPE command forces creation of a MIME-Version header, put the MIME-Version header first. It shouldn't make a difference, but all the other mail clients put the MIME-Version header before any other MIME-related headers so why take chances. send.c. Back to DST creep... Copying my mail file and then writing out a new copy from MM does indeed show all the "From " lines have changed by 5 hours! This doesn't look like a DST difference -- it looks like UCT (GMT) versus local time. Turns out the problem was in ccmd/machdep.h. The definition of HAVE_TZ_SYSV wasn't taking because CCMD_OS_SOLARIS wasn't included among the platforms that select it. ------------------------ Fri Sep 20 11:40:01 2002 Builds: Built on Solaris 2.5.1, Solaris 8, Solaris 9, Red Hat Linux 7.1, FreeBSD 4.4. Some of these required minor tweaks to mm/Makefile and creation of additional config.h-blah files. Failed to build on OpenBSD 3.0: cc -g -DCCMD_OS_BSD44 -c cmgrp.c cmgrp.c: In function rphelp': cmgrp.c:115: structure has no member named 'gr_name' etc etc... The HPUX build bombed right away because CCMD uses ANSI C features but the default cc on HPUX is the bundled non-ANSI one. Changed the makefile targets to require the ANSI compiler and it gets a little farther but still bombs. ------------------------ Tue Sep 24 13:43:54 2002 Added new mm/Makefile targets for NetBSD and OpenBSD that are like BSD44 and FreeBSD but add -Dunix; this fixes the OpenBSD and NetBSD builds because their cc does not define "unix". I have been using MM as heavily as ever since Sept 14th, when I switched to "use-editor-always" and have not lost a single byte. I wonder if I lose that option, will I start losing text again? If so, the culprit is parse()... ------------------------ Sat Oct 5 13:03:06 2002 Linda Wallinger noticed that when you remail a message, the Content-Type header disappears. Further testing shows the same thing happens to the MIME-Version header. Diagnosis: the switch() statement in parse_msg() didn't have cases for the three new headers. Fixed in parsemsg.c. ------------------------ Sat Nov 2 11:20:37 2002 ------------------------ Wed Nov 6 09:46:24 2002 Suddenly about 9:30 this morning, MM started adding From lines that were 20 hours off: From fdc@columbia.edu Tue Nov 5 13:37:57 2002 Date: Wed, 6 Nov 2002 9:37:52 EST From: Frank da Cruz To: ... Spamassasin rejects messages that have non GMT-offset timezones. Fix MM to put dates in RFC2822 format. ------------------------ From Jeff on watsun: while sitting at the command prompt on watsun. mm core dumped There is a core file in ~jaltman/core Here is the back trace: #0 0xef77636c in read () (gdb) bt #0 0xef77636c in read () #1 0xef77ab90 in _filbuf () #2 0x443e8 in cmgetc (c=0xeffff958, fd=0xef7e7248) at ccmdmd.c:226 #3 0x41574 in fill (btab=0x748b4, brk=0xeffff9db "\n") at ccmd.c:854 #4 0x407a0 in parse (fdblist=0x7316c, value=0x7a260, usedfdb=0x7a258) at ccmd.c:437 #5 0x1fc84 in top_level_parser () #6 0x1f4a0 in main () ------------------------ A way for users to fix stale locks? Melissa: "No, actually, MM uses kernel locking over NFS. Contrary to the rumors, it works. Except when the locks get left around." ------------------------ Mon Jan 27 09:45:21 2003 Rebuilt to use /opt/ACISmm/ paths. ------------------------ Date: Tue, 4 Mar 2003 15:25:03 EST From: Eric Weaver To: bug-mm@columbia.edu Cc: eric@columbia.edu Subject: multiple Bcc's not processed Bug-Report: Bug in Columbia MM, version 0.91.0(140) (SunOS) Hi, If one puts multiple "Bcc" headers in their outgoing mail (like when editing the body and headers in an editor), then only the last "Bcc" header gets processed. For example: From: eric To: joe Bcc: bob Bcc: tom will only go to joe and tom. Bob will not get a copy. After composing and getting back to the send prompt, a "display" will show the multiple "Bcc" headers, but only the last one gets processed. Oh, to clarify a related item,... if one specifies multiple "bcc user[, user [, ...]]" commands on MM's command line (like at the send prompt), then that method works ok because MM puts all the users into one Bcc header. /Eric ----- Miscellaneous to do or check: . Subject: mm set dont-type-headers Expand buffer size or max count for this, we need a lot more these days. . Add X-Mailer: header? and/or User-Agent: . Idea: "type [ | filter ]" . Stale locks. Why? MM catches SIGHUP and does the right thing. The problem is that when somebody's dialup connection drops, SIGHUP never comes. . Add an "mm" command that does nothing at top level, and goes to top level if at Send or Read level. . Make X-Spam-Score a known header so we don't have to search the whole text for it. . HEADERS ENTIRE-SUBJECT (as opposed to "subject contains") Ditto for FROM, etc. Maybe make this a switch... /FULL-MATCH ? . HEADERS RECEIVED . "keyword junk keyword vacation" gets a parse error if no messages have a "vacation" keyword. This should just be text, not a keyword. Especially since if it happens in a command file, it prevents the rest of the command file from working (the real error here is that command retry is active when the command source is not the terminal). . Add a command to enable translation of all correctly-tagged messages to UTF-8. Perhaps by automatically choosing a pager like iconv | more. . Where is the mbox From line format specified? RFC 976 (UUCP mail interchange format), section 2.4, "Envelope". Not really. RFC 822 mentions the need for an envelope but does not describe it. MRC says that at least the following formats have been seen, and nothing is written anywhere about what the default timezone is: * From user Wed Dec 2 05:53 1992 * BSD From user Wed Dec 2 05:53:22 1992 * SysV From user Wed Dec 2 05:53 PST 1992 * rn From user Wed Dec 2 05:53:22 PST 1992 * From user Wed Dec 2 05:53 -0700 1992 * emacs From user Wed Dec 2 05:53:22 -0700 1992 * From user Wed Dec 2 05:53 1992 PST * From user Wed Dec 2 05:53:22 1992 PST * From user Wed Dec 2 05:53 1992 -0700 * Solaris From user Wed Dec 2 05:53:22 1992 -0700 Therefore MM, to be general and portable, should support all these formats and be configurable per site (and possibly also per user) about defaulting the timezone. From MRC: "MS Source is in: ftp://ftp.cac.washington.edu/mail/ms.tar.Z It's a baby MM that uses the c-client library. All the basic stuff is there, but it's missing a lot of more advanced facilities of both MM and of the c-client library (MIME, sorting, threading...)." ------------------------------ Thu Oct 9 16:27:15 2003 Date creep, cont'd... This time because of the timestamp in the From line. Columbia's mail agent puts: From fdc@columbia.edu Thu Oct 9 13:35:31 2003 where the time is local. The problem is that when MM rewrites the "From " line, it changes it to GMT. I traced this to mkdate() in mm/dt.c. The #ifdefs did not pick up any of the timezone code in Solaris. Rather than mess with the #ifdefs and risk breaking something else, I simply tacked "|| defined(MM_OS_SOLARIS)" onto the selection clauses that import and check the external timezone variable. This seems to fix the problem: . Wait for a message to arrive in incoming spool directory. . Check its "From " line. . Read the message with today's mm. . Write out the mail file. . Check the same message's From line Now it's the same, rather than 4 hours different. It works the same no matter whether built in Solaris 2.5.1 or Solaris 9. Still Needed: . There is no code to parse GMT +/-hhmm format; I think this could be handled by adding "+0100", "-0300", etc, and the corresponding values to the timezone keyword table. Other Changes: . Added UTC as a synonym for GMT in the timezone table. . Merged solaris-cu and (bogus) cu-solaris targets. ------------------------------ Fri Oct 17 14:55:30 2003 In new_mail() [newmail.c] added #ifdef COLUMBIA..#endif code to default mail spool directory to SPOOL_DIRECTORY/a/b/abc123 if /var/spool/.splitmail exists, where abc123 is any user ID (one-char IDs are handled as .../a/a/a). ------------------------------ Mon Oct 27 13:09:36 2003 Checked envelope date after DST->EST switchover; still OK. Changed default In-Reply-To: value to be the replied-to message's message-id (instead of "Your message of "), which is needed for Remedy and also recommended ("SHOULD") by RFC 2822. ------------------------------ Date: Thu, 22 Apr 2004 7:48:02 EDT From: Eric Weaver To: bug-mm@columbia.edu Cc: eric@columbia.edu Subject: [sys@rg03.ais.columbia.edu: New system errors for rg03 as of 04-21-2004 09:30PM] I think MM's sense of which day is it was messed up by the change to Daylight Saving Time... The message below should be "yesterday", but it is listed when I get a list of messages for "today". Am I missing something? /Eric ---------- Misc ideas not acted upon: Add HEADERS ADDRESS (matching all messages whose to, from, or cc lists contain the address -- similar to HEADERS TO). Print a hint when mail file can't be opened because of lock? What should the hint say? To run some variation of the "killme" script... See: ~/mm/mail.txt message from ron@isri.unlv.edu Idea for MIME charsets... make iconv or whatever part of the crt-filter pipeline. Still have to set terminal charset so iconv can convert *to* it. But then what about replying, etc? Make MOVE and similar command default to "previous sequence" if the previous command was "headers" or other command that created a sequence; otherwise to current message. Change DOWNLOAD to allow a filename as argument, or default to x.eml or something (not mm_msg). Add SET RECENT to define what "recent" means, e.g. last 5 days or "since yesterday". Make "previous-sequence" the default message sequence for COPY, etc. Need "set include-mime-headers always" so I don't have to remember each time. Fix "content-type" to allow specification of HTML, etc. Add HEADERS ADDRESS to list all messages sent from or to the given address or person. Implement the FOLLOW command. Given a message number, it constructs a message sequence showing the whole thread. Depends on In-Reply-To: showing the Message-ID. It can go backwards, forwards, or both. To go forward: find all messages that are In-Reply-To the Message-ID of the given message, then find all the messages that are In-Reply-To the Message-IDs of each of those messages, etc. To go backward, find the message that this message is In-Reply-To. Find all the other messages that are In-Reply-To that same message. For each of these, find the messages that THEY are In-Reply-To, etc. --------------------------------- Fri Aug 26 10:49:57 2005 Downloaded Nelson Beebe's mm 0.94 from Utah. He did a very good job on the makefiles, adding a new top-level one. You can find it in: ftp://kermit.columbia.edu/kermit/mm/utah/mm-0.94.tar.gz I built it on Solaris 9 with "make OS=solaris2.x all". It built OK (but it doesn't with the default OS=solaris10). Started it and it worked OK, but, contrary to what Nelson had said, it didn't have any of my changes. Sent him email repeatedly about this and he never answered. ------------------------ Thu Dec 1 18:09:27 2005 Made crude and minimal modifications to allow MM to use a replacement for movemail that acts as a pop client: .mmint set movemail-path /p/kd/fdc/scripts/pop.ksc init.c: user_init() looks for .mm-use-pop or .cyrustt file in home directory. If found, sets global use_pop variable. The difference is: .mm-use-pop will go to the old pop server. .cyrus will go to the new one. mm.c: After calling user_init(), if use_pop is set, prompts for POP password immediately upon startup. The maximum length is 255. Added a prompt_pw() routine to do this. The password is kept in MM's memory and used every time the "movemail" program is called. newmail.c: if use_pop set, make an argument vector for the POP script rather than regular movemail (first arg is dummy, second is ~/.mm-newmail, as always, and new third argument is password). Skip over code that looks at the mail spool, and always call the movemail program. ------------------------ Fri Dec 2 20:05:12 2005 Changed the default filename for download from mm_msg to mm.eml, for the MS Windows filetype association: print.c. I suppose I should make a SET command for this so we don't discriminate against file associations on other platforms (Mac OS X?) Also updated the copyright year: misc.c. ------------------------ Sat Dec 3 21:53:22 2005 When running MM using "Method 1", in which the POP script replaces movemail, it quickly became obvious that MM would be contacting the POP server way too often, sometimes 2-3 times in succession without a pause, which is not such a big deal if it's just looking at files in the spool directory but we don't want it pummeling the poor POP server. I fixed this by having move_mail() keep track of the time, and never call the POP script more often than 30 seconds apart (only when use_pop is set). When move_mail() is called within less than 30 seconds since the previous POP access it just says "no new mail". Then I realized that MM would try to get new mail only if it saw it sitting in the spool (which won't happen with Cyrus, there won't be any spool). Fixing this required major surgery to new_mail(). ------------------------ Sun Dec 4 15:13:28 2005 Let CHECK-INTERVAL be the minimum interval for contacting the POP server. This way the user can control it. However, MM still won't contact the POP server more than once per 30 seconds. move_mail(): newmail.c. Moved the password prompt to the place where we call the POP script. If the password hasn't been set yet, prompt for it. This way we don't ask for the passowrd if MM is being used (e.g.) to examine some random file. main(): mm.c; move_mail(): newmail.c. But when I did this and then did "mm examine foo.txt", why did it still want to check for new mail? Changed cmd_get() to set a nocheck flag if the file is read-only (EXAMINE) or it is not the user's mail file, and changed new_mail() to check this flag. file.c, newmail.c. But it still checked for new mail. Turns out that in the main program, the call to new_mail() was executed unconditionally the first time through the loop before we parsed any commands and therefore before any mail file was loaded. I changed it to call new_mail() only if there was a current mail file. top_level_parser(): mm.c. I received spam from some place that put junk in the Date: field. This caused the POP script to write a From line with missing date, which MM did not like at all. Fixed the POP script to supply the current date and time in this case. ------------------------ Mon Dec 5 10:27:27 2005 Fixed CONTENT-TYPE command to allow allow the user to specify the entire content-type, not just charset. Discovered that INPUT /CLEAR /NOWRAP has a fatal flaw, namely that if the search target straddles an INPUT buffer boundary, it will never be seen. Worked around this one temporarily by adding a trick to the script. Need to fix it in Kermit. Announced Beta.01 to emailtech-wg@columbia.edu. ------------------------ Tue Dec 6 12:00:38 2005 Don't check spool file if movemail_path is empty, which means that we have some other process delivering mail asynchronously to ~/.mm-newmail (a.k.a. Method 2). ------------------------ Wed Dec 7 10:22:56 2005 Tried to build MM on Linux RH AS4.2, bombs out immediately with ANSI strictness ravings. Looked through my notes and saw that I had not actually made any changes to ccmd except to get it to compile on non-Sun platforms. So it should be safe to just take Nelson Beebe's version. I did that, and of course it built fine on Linux ("make gnu-linux"). When I bring this version back to solaris and build it, it's OK there too, but clashes with my copy of MM because MM refers to some ccmd header files which have been changed to disagree with how some ccmd items are declared in the MM source. Then I copied all of Nelsons's files into the mm tree, and then copied all the files I have modifed on top of those. Then I tried compiling. Each of my files got horrible errors, and I fixed them one by one, importing Nelson's ansifications and modernizations and occasional fixes while taking care to preserve (and sometimes modernize) my changes. This involved about 30 *.[ch] files. Finally MM built OK on Solaris, starts OK, but dumps core if I give it any command. It turns out that the array of function pointers declared towards the bottom of cmds.c is NOT generated automatically; it has to be VERY CAREFULLY edited by hand to match the CMD_blah values in cmd.h, which ARE generated automatically from cmd.H. ------------------------ File pairs like cmds.H and cmds.h, that differ only in case, don't work on case-insensitive file systems like Mac OS X. HEADERS RECENT used to apply to the the messages that came in since MM was started. Now it applies to the ones since we last checked mail. Re-enable locking code via #ifdefs. #ifdef COLUMBIA lock mail file using file locks. Add SET DEFAULT-DOWNLOAD-FILENAME (this name is OK). Ctrl-C... Do you really want to exit MM? [y/n] y <-- doesn't exit. Do you really want to exit MM? [y/n] y Add build instructions. Add list of site flags. Add list of feature flags. Make the POP script remember whether it is logged in so it knows what to send on Ctrl-C. Add this to mm "make clean": rm -f mm argsort mkdefines mkversion mm-trans mmversion sys-prof movemail - prepatch add linux target to ccmd makefile. Message 14 has bad date string (continuing) Message 15 has bad date string (continuing) Message 16 has bad date string (continuing) Message 17 has bad date string (continuing) Here are headers from some of the offending messages: From: Odetta Rogers Date: 16 Dec 2005 16:28:50 -0500 From: Best Postcard Date: Sat, 24 Dec 2005 04:50:16 -0500 (EST) This has happened twice in about 3 weeks. There doesn't seem to be anything wrong with them. Added code to mbox.c to print the bad line, and to substitute the current date/time rather than Wed Dec 31 19:00:00 1969. mbox.c, 24 Dec 2005. Changed Copyright year to 2006, misc.c, 8 Jan 2006. Added QD as an invisible synonym for DELETE, because of 'more' -- first you have to type 'q' to get out of more, then 'd' to delete the message. But if the message was less than a screenful, you often wind up typing 'qd'. cmds.H, cmds.c, etc, 8 Jan 2006. MM writes a bogus in the Resent-Date: header: Resent-Date: Sun, 8 Jan 106 12:42:18 EST The problem with date is in rfctime() in dates.c. Old (which works): sprintf(str,"%3.3s, %d %3.3s %d %d:%02d:%02d %s", day_names[t->tm_wday], t->tm_mday, month_names[t->tm_mon], t->tm_year + 1900, ... New (which doesn't): sprintf(str,"%3.3s, %d %3.3s %d %d:%02d:%02d %s", day_names[t->tm_wday], t->tm_mday, month_names[t->tm_mon], t->tm_year /*+ 1900*/, ... What will I break if I uncomment the "+ 1900"? rfctime() is called in only two places: in writing the Resent-Date (which needs to be fixed), and in sendmail() in sendmail.c, which as far as I can tell, adds the Date: header to every outgoing message. In sendmail(): x->string = safe_strcpy (rfctime ((time_t)0)); In do_remail_one(): msg->resent_date->string = safe_strcpy(rfctime(time(0))); What if I make the second one like the first? Makes no difference. It seems rfctime() is never called when I send a new message because in that case, the date header has already been added elsewhere. I tried to see where, and how, and have the Resent-Date set the same way, but I couldn't home in on it, so I just uncommented the "+ 1900" and everything seems to be fine. rfctime(): dates.c, 8 Jan 2006. The new version of MM seems to rewrite the From: and Date: headers when I remail. Turns out that this happens because the following was added to config.h in the Utah version: /* * Let's let sendmail do it's thing with From:, To:, Sender: headers. */ #define DONT_EMIT_FROM_HEADERS and is easily fixed in config.h by not defining, or by undefining it once defined. config.h, 9 Jan 2006. The other big change since reconciliation with the Utah code is that the mail file is no longer locked, which is bad because it's too easy to start a second copy of MM and then mess up your mail file. Kernel locks are a big pain when using NFS and multihomed hosts (as we do here), but first let's see if we can put it back the way it was... I did in config.h by defining HAVE_F_LOCK within #ifdef COLUMBIA..#endif. The resulting mm binary interoperates with the old one in guaranteeing exclusive access to the mail file and to ~/.mm-newmail. 9 Jan 2006. When I start MM on Solaris, it says "Cannot restore tty attributes: error 9". This comes from ccmdmd.c, the machine-dependent module of ccmd. Error 9 is "bad file number". I see this has been noticed before by other people. It looks like the ccmd routine unraw() is sometimes called on a file descriptor that is not open. I put in a check for this, as well as a check that the tty modes have indeed been saved before we try to restore them. ccmd/ccmdmd.c, 9 Jan 2006. ------------------ How about making a lockfile instead? Then users can fix problems themselves. If MM GETs a mail file in R/W mode, it should create a file in the same directory with ".lock" appended to its name. The lock file should contain: username hostname timestamp pid But then it will only interoperate with itself. Maybe it should do a kernel lock AND a file lock. That way, if kernel locking fails because the file is in use, MM can give the user the info from the file lock.