Nick Mudge Now I want to write an operating system...

I'm looking for the best books in the world on the subjects listed below. They can be beginning books, intermediate or advanced. If you know of other good books on these subjects, that you think maybe should be on this list, please let me know.

Must-Read/Classic

Discrete Math

Computer Theory

Algorithms

Compilers

Hardware/CPU

Operating Systems

Assembly

C Programming

Network Programming

Better Programming

I thought I’d shoot off an idea about programming webpages.

To me writing a script to generate HTML has always been a problem. It’s a problem because you have to mix a bunch of code with HTML. No matter how you mix more than a little bit of code and HTML the result is inelegance, ugly code, hard to read code. It’s not very good because you’re mixing two different syntaxes together in a jumbled fashion. Code that is just code without HTML in it so much nicer and easy to follow and read. And gosh, I’m always perturbed about how to indent code and HTML together so that structure and things can be made clearer. The elegance, clarity and structure provided by indentation is reduced a lot by the combination of programming logic and HTML intermingled, at least from my experience.

There’s been various ways to help make this better. For instance in PHP you can move in and out of HTML and PHP modes which makes the transitions nicer sometimes. The Smarty templating system is largely a system to make this easier. The Smarty language syntax and tools integrate nicer with HTML and help keep programming away from HTML. This also has a nice benefit of being nice to designers that need to work with HTML and CSS but don’t want to hassle with programming.

Smarty is a nice improvement and the best solution I’ve used so far but the problem is still there. Sometimes you want to add quite a bit of logic to your Smarty templates, adding loops, and inner loops, adding lots of if conditions, making function calls etc. You might want to do this because the display of the webpage you are making is complex and requires lots of if conditions and loops or because you want to reuse the same templates for different webpages (this cuts down the amount of work you need to do and manage – reuse is good) and so need to add additional logic that runs depending on which page the template is being used to generate. Too much logic in Smarty templates quickly gets inelegant and hard.

Things get inelegant and harder to follow when you start adding a bunch of logic to HTML pages. It’s like HTML wasn’t meant to have logic in it.

A related problem that is also a big problem with HTML is that it does not scale. The bigger an HTML file gets the harder it is to follow and work with. With a zillion div tags and table tags it gets harder and harder to know what exactly is happening anywhere in a document. It just doesn’t scale. This combined with logic makes for general ickyness.

I wonder if there is another way.

I’ve looked into Lisp a little in the past and what I’ve seen is that in Common Lisp the combination of programming and HTML are handled in a completely different way. I wonder if something here could be a real solution to this inelegance pest, and actually really bring about an elegance and smoothness in the combination of programming logic and HTML.

From what I've seen in Common Lisp when you program webpages you don’t really mix HTML and programming. Instead you make it all programming. I’m thinking it might be something along these lines: You have a library of functions and/or macors. Each function or macro or arguement corresponds to an HTML tag. For instance you have the div function that is used to generate the HTML for a div tag etc. You just use all these functions that generate all the HTML for you. And it’s easy and natural to add logic because it’s all consistent programming code. You keep the consistency, elegance and scalability of a programming language by only programming in the programming language.

The main problem I see with this approach of using functions to generate HTML is that HTML is highly flexible -- some tags can have certain attributes and can be nestled within some tags but not others and there’s a ton of variance and things you can do in HTML. In lots of programming languages I think functions are probably not flexible enough to cope with all the possibilities. I think these functions would limit what you could do with HTML and cut your flexibility of how you want the HTML to be. It really might not be very workable. I certainly don’t like to code with constraints on me.

But maybe it’s possible with something like Lisp. You’d need finely grained functions/macros that had the same flexibility and limitations as the HTML tags. And I think that’s something Lisp might be able to provide. In Lisp you have finely grained control on what your functions are and how they work – more than I’ve seen anywhere. In addition Lisp has powerful macro abilities that enable you to mold the syntax and semantics of the language to your will. This fine grainess of control on the syntax and semantics of the language might enable you to write a library of HTML functions that you could use to write pure elegant code that generates all the HTML you need and with as much flexibility as you need. Perhaps this already exists.

I think something like this would make it much easier and faster to work with webpages because pure well written code is much easier to write and read than a mess of HTML and code together. With code you have one consistent syntax and it scales better than HTML does. In addition if your functions are perfect (have no bugs in their implementation), then any HTML you generate with them should automatically be perfect too – that’s real scalability power when talking about HTML correctness.

Of course this pure programming way of handling HTML might not be so good for designers. So I was thinking that there could be some kind of separate interface for designers – like CSS is a separate interface into the display of HTML and a webpage. One idea is that a designer could design stuff in pure HTML and CSS and then feed it into a program that analyses it and if it is correct HTML and CSS generates or modifies the programming code to generate that HTML and CSS and if it is wrong spits the HTML and CSS back to the designer to fix.


From Chris Minnick:

Good thoughts, Nick.

I see what you’re getting at, but I’m skeptical of solutions to make code or HTML more 'pure'. It seems like no matter what you do, someone is going to start slipping some HTML in where it doesn’t belong, or is going to put something that should be in the CSS into the HTML, or is going to use tables for layout, or what-have-you. Usually, people resist doing things the 'right' way because the right way is more difficult or doesn't work right.

CSS positioning is a perfect example: purists insist that all Web page layout should be done with divs and CSS, and that tables should only be used for tabular data. This makes perfect sense, of course, but the reality is that you just can’t do some of the things you can do with tables using divs, or it’s very difficult—especially considering browser differences.

So, what usually ends up happening as a result of this is that the people who were insisting on purity start backing off … so, now RSS 2.0 is popular, while the strict RDF version (RSS 1.1) was a big flop, and HTML 5 is going to be the successor to XHTML because people continued to use the much more flexible HTML 4.01 years and years after the stricter XHTML 1.0 came out.

My question is: would the benefits of writing HTML using functions outweigh the added complexity and inconvenience to people who design webpages?


From Nick Mudge:

Hey Chris,

I totally agree and excellent point. Your email made me realize that ideally the functions should be flexible enough to allow people to be as non-standard and/or hacky as they want. I think people should have the freedom to do things how they want. I think the functions should prevent obvious errors like a <div> without a </div> and things that will obviously break webpages in web browsers. It would be nice if it could also help people be as standard as they want by letting them set some global setting or something so that functions enforce more strictness to a standard or whatever they want.

I think the goal of a library like this would be to enable programmers to combine programming and HTML in way that is syntactically consistent, easier to read, write and maintain without loosing the flexibility that they want. Or to achieve a scalable elegance with the combination of logic and HTML. Perhaps a secondary goal would be to help people create HTML as correct or as standard/unstandard as they want.

Any system around the functions/macro library such as a program that translates HTML into code that generates the same HTML should be as flexible to the user as the functions.

I think a designer who does HTML and CSS work would use some program that does an HTML to code translation and the designer should have a way to specify how loose the translator is in accepting HTML and CSS. And I think programmers would just work in the code.

I've decided to attempt to write an operating system.

Basically I want to be able to understand a whole system as much as I can and I want to make it as easy as possible for others to understand it as well. This is the main reason I want to write an operating system.

The reason I'm not just studying an existing operating system like Linux or Minix or MikeOS is because I think I will understand a system better if I write it myself and it seems more fun to me. I am of course studying those operating systems to learn to write my own.

Goals and Guides

The design of the system is pretty loose so far but I do have some general guides and goals for the system.

  1. Designed to explicitly take advantage of 64 bit processors.
  2. Designed to explicitly take advantage of multi-core or multi-cpu processors.
  3. Designed to take advantage of and provide virtualization i.e. operating system virtualization etc.
  4. It's a server operating system. Not designed to be a desktop operating system or anything else.
  5. It needs to be designed so that the system can be understood as easily as possible. It must be simple. The code base must be as small as possible.
  6. It must be good.

I'm not sure if the kernel will be a microkernel or monolithic or something else.

Yesterday I watched these two videos: The Free Software Movement and the GNU/Linux Operating System, which is a speech given by Richard Stallman, and Revolution OS, which is a documentary of GNU/Linux.

Below is a quote of Bruce Perens from the Revolution OS video. Bruce Perens created the Open Source Definition. I thought this was pretty funny:

Well I announced open source to the world on the Internet. I did a lot of the early administrative work of starting the open source inititiative. And I think six month later I was reading the words "open source" in the news all the time. I was totally astounded. And a year later I believe Microsoft was talking about releasing some source code. And someone in the press asked Steve Ballmer if they were going to open source their code and Steve Ballmer said "well open source means more than just releasing the source code." And I realized that he had read my document and understood it and was now telling the press about this. Now if you're like just a guy on the net who's not doing this for a job at all and you sort of write a manifesto and it spreads out through the world and a year later the vice president of Microsoft is talking about that, you'd think you were on drugs, wouldn't you? But that's what really happened.

My father was in the third Selma March led by Martin Luther King in 1965. He recently wrote up the experience and I put it on the web: The Selma March Remembered

Ian Lance Taylor has a good piece on why people work on free software.

Vlad Dolezal has a good piece on the real reason we use Linux.

It's interesting to think that all the things you see computers doing is occurring because someone wrote code to make them do it. A parallel world behind the world of applications, email, websites, phones, animation, graphics, systems, processing, transactions, video games -- you name it, is a world of code and programmers.

I think I was originally interested in programming because I was amazed at learning how to make computers do little things. It can be an amazing experience to first understand how to make a computer do something you had no idea about before.

Programming is not a purely mechanical, and analytical activity. It's similar and related to the arts. Sometimes I like to think of it as like writing a story. And when you're done a computer can execute your story, assuming the characters you describe in your code and bring the story to life.

After awhile of programming I learned something about programming even better than being able to make computers do things. I learned this: Like a book that's better than its movie, the code of a program is better than its execution.

Your code is a better thing than its execution. It presents a funny picture. It's funny because so much of the world sees the side effect of code being executed by computers. People see the results of computers executing code. They don't see the code. Sometimes it can make one think that the world only cares about programming and code for what it makes computers do.

Well, let me ask you something. What do books do? What does music do? They don't do anything. You like them for themselves. You might like a good book because it is written really well, or you like the characters or the story line. A computer program is the same idea, you can write it well: have perfect indentation, have good variable names, write concise but expressive code, use the best functions and tools for what you're writing, and use them in the best way. Give your code structure and organization and express it in such a way that the ideas of one section of code flow easily into the next. Write documentation. Have great ideas and express them in your code. In other words write your code so that it is well written and has a great story. If you did this and really cared about it, you might end up really liking your code and realize that its beautiful and a piece of art itself. Perhaps one day you'll snicker when someone demands to know and only cares about what your program DOES.

Of course there's side effects for cared about code. It's easier to understand and maintain, less bugs, its interesting, easier to extend, you actually want to work with it etc.

Of course you are writing code for its execution. I am. I just like and care more about the code than its execution. It's like why you have job. Of course you're doing it for money, but are you doing it only for the money?

Brian Hurt has a nice post about Postgres: Postgres for the win!

Quote:

At this point, the only complaint I have is that Postgres is doing so much with so little that I’ll never get cool hardware to play with. I’ll be stuck with cheap low-end boxes forever. Sigh.

Here's a post he wrote with some great data on Hash Tables: Problems with Hash Tables

Buttons

Newsconomy now has browser buttons so you can quickly submit webpages from any place on the web. A "my newsconomy" button is also available that you can use as a bookmark to your items in Newsconomy.

Sort

The new sort can either sort items by most recently submitted or by most recently traded. It sorts by both by default.

Search

The new search provides a convenient interface to Newsconomy's URL-based tag system to find items.

More items about Newsconomy

I took a little break from programming on my own time but I'm back now. I've recently been studying C and writing some C. I want to write a hash table implementation in C. Here's the general idea of how a string-based hash table works. You have this big regular array with numerical indexes/keys. But you want to use string keys instead of number keys to access elements in the array. So what you do is write a function called a hash function that converts the strings into numbers that are used as keys/indexes to insert, delete, update and retrieve elements of the array. Because hash tables are just using a numerical index into an array, these operations are really fast and don't slow down as more and more data is added.

The first thing I needed was a hash function. I didn't know what to use. So I found some hash functions on the web. But I didn't know how good they were and I wanted to have an idea of how well they worked. A problem with hash functions is that it is difficult to translate all different strings to all different numbers. When a hash function converts two different strings to the same number, it is a collision. You obviously can't have the same numerical index for different array elements, unless you have a collision strategy, which use of reduces performance.

So I wrote this C program to test the 7 different hash functions found on the web. The test generates 50,000 random strings that are randomly between 1 character and 25 characters long inclusive. And each of the random strings are unique -- there are no duplicate strings. The program takes a hash function and runs it on each of the 50,000 random strings and counts how many collisions the hash function produces. I ran the program several times for each hash function. The resulting hash from the hash functions is an unsigned long int, which on my machine is a number between 0 and 4,294,967,295 inclusive.

Here were the results:

hash1: 47,635 to 47,645 collisions
hash2: 415 to 428 collisions
hash3: 350 to 396 collisions
hash4: 0 to 2 collisions
hash5: 844 to 905 collisions
hash6: 845 to 910 collisions
hash7: 0 collisions

The program is pretty slow; it's using linear searches. But it doesn't need to be fast. The real test of the hash functions will be when they are used on real data that the hash table is built to be used with. The collisions might be different with lots of similar strings.

Here's a good tutorial on hash tables.

The title of this post is a bit of a misnomer. It's more like what two things the web is. The web consists of content and indexes. They are both valuable because each cannot exist well without the other. Imagine a dictionary with its definitions distributed randomly everywhere in it with no order. That's what the web is like without links.

Sure a search engine has an index, but that's not what I'm talking about; I'm talking more broadly. A news article with links in is it is both content and an index. Links on a webpage are what make an index, so almost all webpages double as content and indexes or cross-references, however you want to think it.

But some webpages are used just for the index and not for the content. This is what makes search engines so valuable. A search engine is a searchable index.

But search engines are limited. For one thing nothing can replace the understanding of people. And there are things of course that search engines deal with poorly such as javascript, pictures and other multi-media.

One of the exciting things that has been growing on the web is web applications that help people create their own organized indexes -- lists of organized links. These web applications such as del.icio.us and stumble upon, mixx and my own web app newsconomy help automate the process of organizing indexes, sharing indexes, and adding items quickly and easy. But it's human intelligence making these.

One of the amusing things I've found is a lot of people don't get the point of web apps such as del.icio.us, newsconomy etc., and thats fine and amusing and annoying. Well if you are an Internet informationalist, someone who frequently uses and is on the lookout for information on the Internet from various sources, then these human-based web index creation web apps (or some form of this idea) have something to do with you. It's worth your while to find out about them.

One of the nicest songs I've ever heard, from YouTube: No Christmas For Me (Original Song)

More: http://newsconomy.com/tag/youtube

You could use newsconomy for multiple things. You could go there just to look at or find interesting things.

You could find things by different topics by using tags in a url. For instance, say you wanted to find items about dictionaries. You could go here: http://newsconomy.com/tag/dictionary You could find interesting things on youtube: http://newsconomy.com/tag/youtube etc.

When you submit things, you could tag them with certain tags to create lists of items. For instance I created a list of items of my blog posts about newsconomy by tagging them with newsconomy when I submitted them: http://newsconomy.com/tag/newsconomy If I wanted to show someone some blog posts about newsconomy, I would just need to give them the link to the list of newsconomy items.

If you wanted to easily and quickly create an RSS feed with certain items in them, you could make a list of items on newsconomy by using a certain tag, and you would automatically have an rss feed with those items. Here's an rss feed with all items tagged with newsconomy: http://newsconomy.com/rss/tag/newsconomy

You could display rss feeds you make on other websites by using feed2js. I displayed an rss feed from newsconomy on my blog, on the lower right part that starts with "Newsconomy: mudge's items"

When you submit items, they automatically go into your account. You have a list of items that consists of everything you have submitted. For example the user sqs has his items here: http://newsconomy.com/sqs You can get items you submitted (or other people) with certain tags by putting the tags in the url, like this: http://newsconomy.com/sqs/dictionary Putting items into your account and tagging them helps you remember things better, and helps you find them later.

Finally, buying items is a way of trading items with people. It's not fully developed yet. Also, buying items puts them at the top of lists, and all buying is recorded with the items.

One of the things I really like in Paul Graham's interview in Founders at Work is a focus on programming and great software:

What we really thought we needed to do was write more software. We were software guys. Maybe someone who knew more about business would be thinking about going and getting customers, but frankly the idea of customers frightened us. We thought, "Before we go get any customers, why don't we just write a few thousand lines of code?" ... We wrote a lot of software. We thought, "That's what we're good at. That's what we'll do." ... I found I could actually sell moderately well. I could convince people of stuff. I learned a trick for doing this: to tell the truth. A lot of people think that the way to convince people of things is to be eloquent—to have some bag of tricks for sliding conclusions into their brains. But there's also a sort of hack that you can use if you are not a very good salesman, which is simply tell people the truth. Our strategy for selling our software to people was: make the best software and tell them, truthfully, "this is the best software." And they could tell we were telling the truth.

I've been working on a piece of software for some time. I was a news editor for a couple years and have been a full time web programmer for the last six months. My project is an outlet for things I've found on the web, things I've learned and loved in my work, and new ideas.

I released my project very early on because I was hoping that I could get some good feedback on it and some initial users. I got some feedback by asking some people, but in general I haven't gotten as much feedback as I've wanted. Getting some initial users hasn't gone well. fazil is the only consistent user besides myself. I'm not sure what the main reason is that it hasn't gotten a few more initial users. Maybe it just sucks too much. Sure, thanks.

I haven't promoted it much because I haven't felt that it was ready yet (and I feel like promoting it a lot right now wouldn't do much). But it is ready for some users now. It is useful now. I love what it does now and I use it. You can use it for some of the same things you'd use del.icio.us for. It has some similar uses as del.icio.us but it is also different.

I am worried about users. I want to make something people really want. So I'm going to write a few thousand lines of code. I'm going to keep adding features, make it look better and make it more understandable.

When things are tough, write some more software.

I love how Joshua Schachter describes describing del.ico.us, from Founders at Work:

It is a challenging product to do conceptually. It's not something like, "Let you file your taxes better." There's no clear value proposition here. It is valuable, but hard to understand. You will be able to remember more things this way, and with that, people don't even realize there's a problem. So that's a challenging value proposition to explain or get across. Ultimately, I think people who understand it are better for it, but it's a challenge.

My project is newsconomy.com

It feels like it has been a long time since I last wrote an entry, and I guess it has been awhile, I suppose in blog time. It's not that I haven't had some good stuff to blog about, I have. I think it's like I haven't been inspired at the right time when it was good to blog.

I'm inspired tonight. I'm inspired by quarterlife. Artistic! and I don't use exclamation marks lightly.

I originally came across the quarterlife show while I was cruising on YouTube. I watched every quarterlife episode there was on YouTube at the time. It's a well filmed artistic show — about artists really and life. Well I came across it again on YouTube tonight but this time I explored past Youtube and watched more episodes on the quarterlife website.

One of the things that really fascinates me about quarterlife is that it isn't just a show. I mean, you can tell that it is a show for the sake of showness, artistic showness particularly. But quarterlife is also a social networking/video/art website &mdash which is also in the show sometimes. The show made me understand and want to do quarterlife. I wondered if the people behind it were doing it for the show or the social network, but now I think for both. They match like a glove.

The quarterlife about page is pretty good.

For some time I've tinkered with the idea of making videos and video blogs and such. I'm tipped over the edge now. I signed up with quarterlife and hope to be posting some videos there pretty soon.

quarterlife on newsconomy

Recently my step dad pointed me to the website of the company where my aunt Teresa Elam works. Interesting to know how involved my aunt has been with open source software. She has a nice reference in the company history, getting FreeBSD out into the world. She's also worked with Patrick Volkerding, the guy behind Slackware, and appears high on a list of Gentoo contributors.

Chris Minnick, the owner of Minnick Web Services, where I work, has been writing some articles on internet evolution. His latest is Is the Semantic Web Doomed?.

I read some good articles tonight. The article So you want to be a consultant...? is really great.

Does it ever happen to you that you read something really good and it has a big effect on you? I read the above article tonight and that's what happened to me. It changed the way I think, made me smarter, better, and more good.

Here's a few other good articles I read tonight:

My list

What if you want from a news/bookmarking website all recent items that have been tagged politics or government or ronpaul? What if you want recent items tagged with both web and design? What if for some weird reason you wanted all recent items tagged with politics or government or ronpaul or web and design? What if, what if? Newsconomy has this now.

Check out these links:

With this tag Kung-Fu under your belt you could probably make any kind of list of items on the web that you wanted.

RSS? Of course. Every list of items on Newsonomy has a matching RSS feed.

More info about Newsconomy.

Newsconomy now has an RSS feed for every webpage that has a list of items on it.

Here's some of the feeds:

What can you do with rss feeds? A lot! Your imagination.

One thing you can do with rss feeds is use them to display items on other websites. For instance, if you look at the lower right part of my blog you will see a section that starts off with "Newsconomy: mudge's items". Below this heading is a display of my newsconomy rss feed. Any time I submit an item to Newconomy, this list on my blog is automatically updated within an hour. I created this display using a free service called Feed to Javascript. You go here, submit the rss feed you want to display, and then Feed to Javascript will create the javascript code that you will then just need to paste into the web page where you want the feed displayed. No coding or fiddling with XML required.

Facebook has an import rss feed/blog feature that I used to import the Newsconomy homepage rss feed onto my facebook profile which you can see if you look at the section of my profile entitled "Notes" on the left side.

Various improvements have been made to Newsconomy.com in the last few days.

Today I released several new ones:

  1. No more browser security warnings. I bought and installed an SSL certificate which authenticates the identity of the server and eliminates the browser warnings.

  2. No more categories. I've changed Newsconomy to be tag based instead of having categories. So when you submit an item you make up or choose what tags you want to associate your item with. Right now items can have up to 5 tags. Example: To see all items tagged with programming, you would go here: http://newsconomy.com/tag/programming, for items tagged with design: http://newsconomy.com/tag/design etc.

  3. I redesigned and rewrote the item submission system. Submitting an item is different, easier and better.

A lot more functionality and improvements are on the way. I've been getting some good feed back and suggestions from some people which is great.



My Sites
Links
Archive
Past Blogs


Valid XHTML 1.0 Transitional