Thing-a-day update for newLISP on Rockets 2.0

Blog post

Author
Message

Rocket Man
Posts: 371
Posted on: 2018-09-07 14:46:15.000

If you haven't yet, read my previous post here to get the background information: http://newlisponrockets.com/rockets-item.lsp?p=123

This thread is where I will update all new progress on newLISP on Rockets 2.0.

Rocket Man
Posts: 371
Posted on: 2018-09-07 14:47:00.000
Thursday, September 6, 2018: Created new AWS instance, installed Ubuntu 18.0.4 LTS, installed Rockets from scratch, fixed some bugs in the installer

Rocket Man
Posts: 371
Posted on: 2018-09-07 14:48:41.000
Friday, September y, 2018: Posted blog post on newlisponrockets.com to explain the basic plan and mission. Updated files in GitHub to include missing images and Rockets captcha images for registering new users.

Rocket Man
Posts: 371
Posted on: 2018-09-10 13:19:40.000
Monday, September 10, 2018: Fixed install script to work on Ubuntu LTS 18.0.4. Added Eniac picture to GitHub repo. Fixed bug preventing forum page from loading when not logged in on newlisp 10.7.

Rocket Man
Posts: 371
Posted on: 2018-09-11 15:37:54.000
Tuesday, September 11, 2018: Installed FTP on new AWS test server, fixed bug that prevented Admin page from loading.

Rocket Man
Posts: 371
Posted on: 2018-09-12 14:48:04.000
Wednesday, September 12, 2018: Pulled out the top menu navigation definitions from the navbar partial file into a separate config file (Rockets-navigation.lisp). Added a form to the Admin page to edit the site name and menu navigation options (non-functional right now but displays navigation names and destinations in the form)

Rocket Man
Posts: 371
Posted on: 2018-09-13 15:36:18.000
Thursday, September 13, 2018: Added the ability to modify site name and top menubar link names and destinations in the Admin menu. Updated installer script to set permissions to make this possible out of the box.

Rocket Man
Posts: 371
Posted on: 2018-09-14 13:15:20.000
Friday, September 14, 2018: Added "Add item" and "Delete" buttons to the Admin page's menu config utility. Now you can configure your main menu to be whatever you like, with as many items as you like (at some point, too many menu items run off the page, though)

Rocket Man
Posts: 371
Posted on: 2018-09-17 13:03:48.000
Monday, September 17, 2018: Added selection of front page style in the Admin configuration panel. This will allow you to set the look of your front page. Choices are:

* Single page with custom content
* Single page with blog posts
* Two columns with custom left hand navbar content
* Three columns with custom left and right hand navbar content

Currently this doesn't actually do anything to the main page, but it will save your choice in the Rockets config file, and the default setup script is now updated to include this config value.

Rocket Man
Posts: 371
Posted on: 2018-09-18 11:37:26.000
Tuesday, September 18, 2018: Updated main page to reflect the configuration set in the Admin panel yesterday. Added placeholder left hand and right hand navigation panels in /partials/, will fill in these later.

Rocket Man
Posts: 371
Posted on: 2018-09-19 11:12:41.000
Wednesday, September 19, 2018: Added feature to the Admin page to let admin users upload a new header image for the blog.Can be any image at all, saved to the /images/ subdirectory. Updated the installer script to set proper permissions for this directory.

Rocket Man
Posts: 371
Posted on: 2018-09-20 11:15:50.000
Thursday, September 20, 2018: Moved the custom front page into a partial file (partials/rockets-custom.lsp), added the ability to change the long name of the blog in the Admin page, fixed a couple of bugs with Polls in posts/forums (fixed the bug where a blank 'Vote' button was added to a non-poll post, then fixed some bugs that cropped up as a result of fixing that bug, removed old Rockets logo from the 'you already voted in this poll' page)

So just some little things, but it's starting to take shape. Tomorrow I want to start filling in the left-hand and right-hand panels.

Rocket Man
Posts: 371
Posted on: 2018-09-21 10:40:45.000
Friday, September 21, 2018: Okay, so it's Friday and it's been a long and productive week. So the update today was just a new logo for the blog, to signify the new effort to create Rockets 2.0. Because of the work I've done this week and last week, I've bumped the version number to 0.98.

Sometimes it's the little things!

Rocket Man
Posts: 371
Posted on: 2018-09-24 13:45:12.000
Monday, September 24, 2018: Added multi-part checkboxes for configuring the left hand and right hand admin panels to the Admin page. In doing so I uncovered a bug when using ($POST) for multi-part check boxes in forms-- it works differently than for ($GET). Updated the Rockets documentation to clarify this. I also bumped the version number to 0.99.

Rocket Man
Posts: 371
Posted on: 2018-09-25 11:33:51.000
Tuesday, September 25, 2018: Fixed the bug I found yesterday in the ($POST) function for multi-part web forms (re-updated the docs to indicate that it works now). The admin page now saves the left-hand and right-hand config panels to Rockets-config.lisp. Bumped Rockets version to 0.991 for this fix.

Tomorrow I'll work on actually getting the left hand and right hand panels to display things that the user has configured!

Rocket Man
Posts: 371
Posted on: 2018-09-27 14:58:40.000
Thursday, September 26, 2018: Okay, so it wasn't tomorrow after all but today I updated the Admin page so that it will preselect all checkboxes that you've saved for the left hand and right hand panels, and put in stubs in the panels themselves for each component.

Rocket Man
Posts: 371
Posted on: 2018-10-09 15:16:50.000
Tuesday, October 9, 2018: Back from a short vacation, I'm working on the left hand and right hand panels!

Rocket Man
Posts: 371
Posted on: 2018-10-10 13:47:22.000
Wednesday, October 10, 2018: Added Popular Blog Posts and Recent Forum Posts sidebar functionality, fixed posting bug in Forums, added polls to Forum posts.

Rocket Man
Posts: 371
Posted on: 2018-10-11 13:06:44.000
Thursday, October 11, 2018: Added PostTags to the database, added database upgrade script and modified setup-rockets script, added code to display all tags in the sidebar, added UI for entering tags on new blog posts.

Rocket Man
Posts: 371
Posted on: 2018-10-16 11:51:37.000
Tuesday, October 16, 2018: Fixed tag saving to database, fixed code to display tags in sidebar, testing

Rocket Man
Posts: 371
Posted on: 2018-10-17 14:54:31.000
Wednesday, October 17, 2018: Added multi-tab UI for admin panel, added custom HTML boxes to the Custom tab, added blue button option to Rockets library.

Rocket Man
Posts: 371
Posted on: 2018-10-18 13:21:22.000
Thursday, October 18, 2018: Added custom HTML panel partials and raw HTML files, admins can edit these in the Admin panel now.

Rocket Man
Posts: 371
Posted on: 2018-10-31 18:01:34.000
Hey folks! I haven't stopped working on Rockets, but I'm taking a few weeks off to complete a Machine Learning course. I'll be back!

Rocket Man
Posts: 371
Posted on: 2019-01-03 11:03:20.000
Thursday, January 3, 2018: Happy New Year! Today I got back to work on Rockets, adding a Media tab to the Admin page- can currently view all image files in /images

Rocket Man
Posts: 371
Posted on: 2019-01-07 11:13:02.000
Monday, January 7, 2019: Added file uploading to media tab in admin, added fileupload.lsp to git, added ability to delete images and confirm deletion.

Rocket Man
Posts: 371
Posted on: 2019-01-08 11:52:56.000
Tuesday, January 8, 2019: Added User Configuration page, displays all user info and adds UI for deleting users (can't delete users just yet)

Rocket Man
Posts: 371
Posted on: 2019-01-09 11:28:37.000
Wednesday, January 9, 2019: Added deleting user records (except first admin user!), fixed bug with new users logging in for the first time and fixed a bug with logging in with the new version of newlisp.

Rocket Man
Posts: 371
Posted on: 2019-01-10 14:10:46.000
Thursday, January 10, 2019: Added "individual page config" to Admin panel, letting you choose a layout for individual blog posts or pages. Modified setup script to add this new config item.

I'm going to take tomorrow off. Next week will be moving off the Admin page and changing the way individual blog posts, pages, and forums look. Going to make the forums more mobile-friendly and add the ability to edit comments.

Rocket Man
Posts: 371
Posted on: 2019-01-17 13:07:43.000
Tuesday, January 15, 2019: Added ability to filter posts by tags on the main page.

Rocket Man
Posts: 371
Posted on: 2019-01-17 13:36:43.000
Thursday, January 17, 2019: Altered Posts database to add PostLastAuthor and PostLastDate, updated the upgrade.lisp script and the setup-rockets.lisp script.

Rocket Man
Posts: 371
Posted on: 2019-01-21 11:40:19.000
Monday, January 21, 2019: Added new function display-responsive for responsive tables, changed forum display to use this feature instead of display-table. Still tweaking the appearance, but it flows on mobile now!

Rocket Man
Posts: 371
Posted on: 2019-01-22 15:13:03.000
Tuesday, January 22, 2019: Added parameter for display-responsive to put in a numerical list that specifies widths for all columns. Working on editing the CSS to add striped colors and header backgrounds (haven't promoted these to the main site yet)

Rocket Man
Posts: 371
Posted on: 2019-01-23 14:27:12.000
Wednesday, January 23, 2019: Added striping to responsive table display for the forum, tweaked colors.

Rocket Man
Posts: 371
Posted on: 2019-01-29 11:27:40.000
Tuesday, January 29, 2019: Changed forum view for threads to be responsive, altered Posts table in database to have Last Post Date be a DATE field rather than INTEGER.

Rocket Man
Posts: 371
Posted on: 2019-01-30 11:14:05.000
Wednesday, January 30, 2019: Modified database upgrade script to add data for Last Author and Last Reply Date to all posts. Now we can sort the forum by last posted date, so new replies will always show up at the top!

Rocket Man
Posts: 371
Posted on: 2019-01-31 14:30:17.000
Thursday, January 31, 2019: Added automatic updating of Last Post Author and Date, and added sorting of the forums by Last Post Date. Some debugging.

Rocket Man
Posts: 371
Posted on: 2019-02-04 13:26:57.000
Monday, February 4, 2019: Fixed bug that could have allowed people to edit posts without being logged in. Working on adding editing of comments in threads (have to fix errors like the date above!)

Rocket Man
Posts: 371
Posted on: 2019-02-05 11:23:56.000
Tuesday, February 5, 2019: Added editing of comments in both forum view and post view. Only admins can edit comments for now.

I went back and fixed some of the broken comments (missing bold end tags) and fixed the wrong dates. So it works!

Rocket Man
Posts: 371
Posted on: 2019-02-11 14:58:05.000
Monday, February 11, 2019: Busy week last week, but I'm back on adding features.

Today I added an anchor link to the post box on main page, added selection of multiple post types (Blog post, Page, Comic, Podcast) on main page, added display of Post Type in forums and on main page.

Right now the Comics and Podcast links don't do anything other than create a new post with a different post type. Comics and Podcast support are coming soon!

Rocket Man
Posts: 371
Posted on: 2019-02-13 11:49:08.000
Wednesday, February 13, 2019: I'm working on Comics support (have the barebones stuff ready but still need to add image uploading for Admins and upload the various navigation images) but in the mean time, I fixed a couple of bugs today:

1. Issue with registration failing if you've deleted users in the Admin panel. Silly bug-- turns out that counting all registered users and adding 1 to the number doesn't work if you delete people! Switched it to getting the MAXimum user number and adding to that

2. Fixed a long-standing issue with LastPass on the Registration page by reordering and renaming the fields. Makes more sense now for new users.

Rocket Man
Posts: 371
Posted on: 2019-02-14 15:29:59.000
Thursday and Friday 14-15, 2019: No updates for today and tomorrow, although I did fix a new bug that I introduced with editing comments.

Comics support is pretty close, then it's just Podcast support and support for multiple sites, then I'm going to import the database from my existing blog (jeremyreimer.com) and see if I can get it running on Rockets 2.0. Not sure if I'm going to actually implement the multi-site setup because I have a manual way of doing it (documented here on this site!) that I can set up with editing a single Apache config file.

So I'm getting pretty close to finishing! I might add a few things (like https support) to the new instance, then some testing, then it's on to creating the content for redplanet.mars, the reason for doing this upgrade in the first place!

I'm excited!

Rocket Man
Posts: 371
Posted on: 2019-02-19 11:47:29.000
Tuesday, February 19, 2019: Finished comics support! I don't have any comics on the Rockets blog, but I do have them on my personal blog. I tested the functionality on my test install, but the real test is when I import all the comics in from my personal blog to my new Rockets-powered blog.

Podcast support will be interesting because I have a kind of jury-rigged podcast XML creation engine on my old blog right now, so it might take some work porting it over. I'll be talking more about it later.

Rocket Man
Posts: 371
Posted on: 2019-02-28 10:52:08.000
Thursday, February 28, 2019: I took a bit of a break, but now I'm back and working on one of the last major sections of the blog! This is a podcast section, that will support multiple podcasts and generate an XML file consumable by iTunes and all other podcast aggregators.

Today I added a new "Podcast Configuration" section to the Admin panel and set up a default set of config options for podcasts.

Rocket Man
Posts: 371
Posted on: 2019-03-04 11:12:23.000
Monday, March 4, 2019: Fixed database issue with sorting of PostViews in the most popular posts list.

Rocket Man
Posts: 371
Posted on: 2019-03-05 14:26:01.000
Tuesday, March 5, 2019: Built the front end UI for editing podcast configurations.

Rocket Man
Posts: 371
Posted on: 2019-03-11 14:33:08.000
Monday, March 11, 2019: Finished the back-end saving UI for podcast configurations-- can now add multiple podcasts and edit the settings for all of them. Now just have to build the part that generates the XML file, and it's basically done!

Rocket Man
Posts: 371
Posted on: 2019-03-19 14:24:02.000
Tuesday, March 19, 2019: Finished the part of the podcast script that generates the XML file. Still some minor tweaks that need to be done involving uploading the .mp3 file itself and then re-running the XML generation when editing to add the name of the .mp3 file to the XML entry. Still, this was a big piece of work that I'm glad to have finished.

Rocket Man
Posts: 371
Posted on: 2019-03-26 13:38:25.000
Tuesday, March 20, 2019: Added a "Download" link to the main Rockets page, fixed some image uploading bugs, next step is to allow .mp3 uploading for Podcasts.

Rocket Man
Posts: 371
Posted on: 2019-03-28 13:33:18.000
Thursday, March 28, 2019: Added ability to add .mp3 files with the [ audio ] tag (administrators only). Right now it uploads to the /images/ directory which isn't ideal but I'll fix that later.

My old personal blog used to auto-replace these files with a link to a mini-Flash player. The reason I did that and didn't use the brand-new HTML5 <audio> tag is that, back then, the tag would actually crash your browser if you had too many audio files on one page (and since my blog could display all podcasts at once if you wanted, that was a problem) I even wrote a blog post about it where I complained that HTML5 sucked, which ended up being one of the most read posts on my blog.

But time goes on and heals all wounds, as it were, and the tag no longer sucks, and HTML5 no longer sucks in general, and Flash is dead. Like completely dead. So, progress...

Rocket Man
Posts: 371
Posted on: 2019-03-29 11:21:00.000
Friday, March 29: Added BBCode for embedding URL links with a description. It works like this: type in [ url = http://test.com ] link in here [ / url ] (without the spaces)

Let's test it right now! link to test.com

xerxes
Posts: 103
Posted on: 2019-03-30 11:52:52.000
Amazing progress. Not long ago, maybe 2018, i cast a glance at this site, concluding development had halted, RocketMan too busy... sort of thing.

Now, on return, finding all this work done, i have to congratulate RM on all this wonderful, useful, brainy stuff. Thank you!

Rocket Man
Posts: 371
Posted on: 2019-04-03 09:13:06.000
Thanks, Xerxes! I hope I can continue to have time to update Rockets even after 2.0 is released. It's kind of nice building a thing and then making it grow over time.

xerxes
Posts: 103
Posted on: 2019-04-20 04:04:41.000
Just a thought:

Is it easy to add a button to return to the top of the (same) page? When there are many posts, a lot of scrolling is needed ...

Rocket Man
Posts: 371
Posted on: 2019-04-23 15:50:38.000
Yes; I have a button to jump to the post reply box, and I can add another to jump to the top. Of course the "Home" key does the job as well. I use it a lot when browsing!

Rocket Man
Posts: 371
Posted on: 2019-05-02 16:16:12.000
Just a quick update-- I'm a bit bogged down at work, and so progress has slowed a bit. I'm also doing a database migration of my old database (from my personal blog) to Rockets, and it's going more slowly than I anticipated-- it's very finnicky, with just SLIGHT differences in places for each table.

This is an artifact of me building my blog (Monarch) first and then building Rockets afterwards and fixing a bunch of mistakes I made the first time.

Anyway, I'm chipping away at it day by day and when it's done it will be a good test of Rockets 2.0 being able to do everything my old blog did (and then some) and I can get back to adding the last few features and testing before the release of Rockets 2.0.

xerxes
Posts: 103
Posted on: 2019-05-03 00:07:25.000
OK, best wishes with all that!

Rocket Man
Posts: 371
Posted on: 2019-05-10 15:50:02.000
Small update: I did the first full migration of the old (Monarch) database to Rockets 2.0. It came out about the same size which was a good start. I swapped it into my test instance of Rockets and unfortunately it broke when displaying the main page due to some incompatibility with poll formats. But when I commented out two lines to stop parsing polls, it worked fine, and I was able to see all my old posts in the new framework!

Next week I'll be fixing these issues (probably the format of polls is slightly different, or there's an edge case where there are no polls on a post that isn't being handled the same way) and then I'll fix any other small issues that come up. Then I'll be able to keep testing Rockets 2.0 with all my old content from my website, which will help quite a bit with the overall testing effort.

Just one more feature to write, and that's RSS feeds. I don't expect it to be that difficult or take that long. Then a lot of testing, and writing some docs, and Rockets 2.0 can finally take off!

Rocket Man
Posts: 371
Posted on: 2019-05-14 14:52:16.000
Fixed the issue with the database migration and the polls error.

I also changed the way error messages worked with failed logins. Now it lets you know if your user name or password is incorrect instead of just saying that one of them is wrong. Should add some flood control to prevent brute-force guessing.

Rocket Man
Posts: 371
Posted on: 2019-05-14 15:28:46.000
Fixed issue when a user had admin privileges but was not user#0 and could not see the link to the Admin page in the dropdown menu.

Rocket Man
Posts: 371
Posted on: 2019-05-17 13:56:08.000
Made a major breakthrough in my database migration-- my old database from jeremyreimer.com now imports with no errors when running on Rockets 2.0. Still have to fix some formatting issues (apostrophes became &lquot;, missing line spacing) but other than that it's good to go!

While writing the database migration feature I ran into a problem using Rockets' built-in (create record) macro. It's really easy for adding new records to the database, but newLISP and SQLite are slightly different in the way they handle nulls. In newLISP, a null value is called 'nil'. In SQLite, a null value is created by NOT adding a value to a column in the first place. You cannot assign a "null" value to a column in any way if you use that column name in a SQL statement.

Normally this is not a problem, but it became an issue doing the database migration, where I was reading values out of one database, getting the "nil" value in newLISP, and then trying to create a new record with that nil value in any column. Instead of making a null value in the database it made the literal string value "nil" in the database. I thought about updating my script to work around this, but then I realized I could fix it in Rockets itself.

Long story short, the (create-record) macro now lets you pass in column names with nil values, but if it finds these values it just removes them from the SQL statement that it passes to SQLite. This way the null value is created properly in the database.

Kind of a tricky problem, but one that was very satisfying to fix. I bumped Rockets' version number to 1.95 just to celebrate. We're getting very close to 2.0!

Rocket Man
Posts: 371
Posted on: 2019-05-22 11:45:56.000
I've fixed all the import issues with the database-- there were a bunch of them!

But I still have to convert the old HTML code that I put into a lot of my blog posts. My old blog allowed HTML code in posts, which can be a bit of a problem with things like cross-site scripting attacks. You can try and get around this by removing things like the <script> tag but it's a bit tricky.

Rockets (both 1.0 and 2.0) prevents such attacks by refusing to parse HTML code at all-- the < and > characters are translated into literal < and > characters and rendered as such, instead of letting the browser render them. Instead I allow what is known as "BBCode"-- using square brackets to do things like bold and italics and such.

So I'll have to do the translations to make these parts of the posts show up properly from my old blog. Also, I plan on moving over a handy BBCode guide and inserting feature into Rockets 2.0.

Rocket Man
Posts: 371
Posted on: 2019-05-23 15:18:14.000
Minor change to Rockets based on my database import-- I changed the way the main page displays so that it includes post types of both "Blog Post" and "Podcast" in the main stream.

I figure anyone who has a podcast probably wants the podcast to show up in the blog, so I'll leave that in for now.

Rocket Man
Posts: 371
Posted on: 2019-05-27 15:45:49.000
ok, this is a very simple little bug fix, but it's something...

Rockets automatically converts url links to hyperlinks like this: https://google.com

But now it disables it if you use the [ img ] tag, like this:



The problem was that the hyperlink was getting converted to a clickable link first, then the [ img ] tag tried to turn it into a <img src='whatever.jpg'> tag and the combination of the two was just a broken mess.

I found this bug when I was trying to convert my old blog posts, which had direct <img src=....> built in, to the [ img] tag. So actually it's turning out to be quite helpful to do this slow, tedious migration in terms of testing!

Bumped Rockets to version 1.96 just so I can keep track of when I've applied this fix.

Rocket Man
Posts: 371
Posted on: 2019-06-05 17:14:11.000
Moved Rockets to a new server, bumped version to 1.98. Downloaded a fresh copy of Ubuntu Server 18.04 so I can test the one-button install script.

Rocket Man
Posts: 371
Posted on: 2019-06-05 21:06:20.000
By the way, I finished the migration of my personal site, http://jeremyreimer.com, to the latest version of Rockets. It's also live on the new server.

Rocket Man
Posts: 371
Posted on: 2019-06-06 22:22:47.000
Okay, so I set up a brand new VM and downloaded a bare-bones Ubuntu 18.04 LTS straight from ubuntu.com so that I could test out the newLISP on Rockets install script!

Since the new installer doesn't let you set up a LAMP stack, I added an Apache 2 install in the script. If you already have Apache 2 installed, it's fine, it will just skip installing it.

I also had to add another symlink for that troublesome libcrypto.so.1 library, since it moved around AGAIN compared to Ubuntu 16. I've left the other symlinks in, so in theory this install should work with Ubuntu 14, 16, or 18. I had to fix the permissions as well so that you could post to your new database, and I had to add the new fields PostLastAuthor, PostLastDate, and PostTags to the sample post that gets inserted into the database.

I've made these adjustments to the setup-rockets.lisp script, which gets run automatically at the end of the install-rockets.sh script. I'll test it one more time with a brand new VM just to make sure. I'm using this VM as a development environment and for updating Rockets in GitHub, so I can get rid of the old AWS server that was both the production server for Rockets and the place I updated stuff in Rockets on GitHub from, which was weird.

Rocket Man
Posts: 371
Posted on: 2019-06-10 21:23:58.000
Added the ability to customize the image on the Forum page, and add a custom forum subtitle.

Moved the positions of the main blog and forum image uploads to the top of the first Admin page.


Rocket Man
Posts: 371
Posted on: 2019-06-11 23:08:18.000
Added a link on the "Last Post" column that jumps you to the last post in a thread. Only works if you're signed in.

Rocket Man
Posts: 371
Posted on: 2019-06-12 17:16:04.000
Changed "New" icon so it is one of a pair of icons-- "read" and "unread" that look similar. This way new users don't see tons of "New" icons everywhere.

Also updated the way the main forum listing is displayed so that it will look better on mobile.

Rocket Man
Posts: 371
Posted on: 2019-06-13 18:25:12.000
Modified the 'new' and 'read' icons slightly, and made it so if you aren't logged in you still see the icons (but they are just all 'read')

Added "Forum notices" as a post type that Admins can post from the front page. They don't appear on the front page-- they appear with a diamond exclamation icon but only in the forums. They appear at the top of every forum page. This is a good place to put announcements and notices that are related to the forum itself.


Views: 1082