A Web Luabrary

[technosophos]

If we were to provide a full web library for the Lua engine, what sorts of things would we include? Here are my thoughts:

• We might need to add the ability for a Lua file to load a file that has Lua-In-Markup (<?lua...)
• One huge thing we should do is provide a caching service to cache data between requests.
  • Caches could be scoped per app (as configured in Caddy), and they would persist for the duration of the app.
  • Caches would be expiring key/value pairs (with the option to set expiration to never). Behind the scenes, the data could be serialized.
• Support for modifying pieces of the HTTP response is required. Examples:
  • Ability to change the HTTP Staus code
  • Add response headers, especially content-type
  • Possibly signal the handler to send back data in chunks (Data Frames in HTTP/2, Chunked in HTTP 1.1)
  • Simple support for some frequent needs, like sending redirects.
  • Force buffer flushes of output data?
• Support for getting certain pieces of the request
  • Fetching the method (GET, POST, etc), protocol, and so on.
  • Simple way of determining whether the connection is over SSL/TLS
  • Access to host name (preferably a simple way that does the header checking for you) and destination port
  • Access to any sent headers
  • Access to URL parameters
  • Simple access to POST form values, regardless of post encoding
  • Simple access to raw POST data
  • The ability to handle non-standard verbs like PATCH, PROPSET, etc.
• HTTP specific add-on functionality
  • A nice session abstraction
  • A nice authentication abstraction that supports both HTTP-based authentication and token-based authentication (or, ideally, is more generic)
  • A top-qualtiy form abstraction that provided the following features:
    • Generate forms in code
    • Add security characteristics, like form tokens
    • Cache forms on the server
    • When a form data is submitted, extract the cached form and rebuild the form object so that the developer is working with the same data structure she/he defined in the first place. This extraction provides a place for validation and security measures, too.
• Data storage services
  • Provide a way to cache handles (or connection pools) to things like databases
  • Provide access to tempdir storage with automatic cleanup?

[mholt]

Excellent ideas. I've been stewing over similar things today. And I like the name "Luabrary" haha.

To add to your list, there should also be a good way to produce and consume JSON and (eventually) other common serialization formats.

As far as data persistence is concerned, this is an area where PHP is kind of weak and it would be nice if this Luabrary was a little more helpful. The session abstraction you talked about is probably essentially a key/value store except scoped to a particular user. (It would be nice, too, if sessions were implemented to automatically mitigate session hijacking.) What defines the lifetime of the app? Just however long Caddy is executing? Or do we dump to disk and if so, how often? Or maybe it can use an external data store like a database. (This is not a simple task! Lots of effort will be required here.) Go has some data stores that are pure-Go and in-process, which is really appealing. Maybe we could look into one of those.

I'm gonna play around to see what it takes to do simple things like convenient access to the HTTP response and request, and if I like the result, I'll push it.

[technosophos]

I think a pure Go in-memory store might be really nice. I don't think this kind of storage would have to persist across a reboot, so in-memory should be good.

For "lifetime of the app", I was thinking generally that would be however long Caddy is executing, though I figured we might at some point want to make it possible to restart a Lua app independently of a web server restart (like, say, for dev mode).

I have been trying to refine my idea of a shared cache, and so far this is the best I've come up with:

• It would be nice to have a secure session system that also allowed storing key/value pairs in a session object. These would last for the duration of the session itself.
• Offering a global key/value store would be great for an entire class of things. For example, it would be great to be able to use it as a page cache or an asset cache.
• What would be AWESOME (though probably a version 2 feature) would be to make the caching back-end pluggable, so that a cluster of Caddy servers could, say, share a Redis or Memcache store instead of using the local in-memory copy.
• Some things, like database connections, would be great to persist for the entire lifetime of the app. But they are maybe a little more complicated to persist in a key/value store because the "value" is a connection. That said, PHP offered a persistent DB connection, and hardly anyone uses it. So maybe this class isn't as big of a deal -- especially for a version 1.

Finally, I was thinking more about the possibility of using concurrency inside of Lua, and realized that if we allow that, we should make sure that things that we pass in are either read-only (like some of the request data) or are protected by a mutex.

[mattfarina]

The name "Luabrary" made me chuckle. I like it and the list of features is a good start. Here are some more that may be useful:

• Cookie handling. I think you'll need this for sessions, too.
• For HTTP/2 connections the ability to push assets that have not been requested, yet.
• Easy serving of file assets (CSS, JS, images) from the local filesystem or other location (e.g., s3).
• A template that can handle inheritance engine?
• The ability to do buffered output and chunked output (not knowing the final output at the time you start serving the HTML).

There are two ways I've seen PHP used to build things for the web. One is as tags inside HTML and it makes local modifications. The other is as a full web programming language. Is the idea of Luabrary to handle both?

[mholt]

FWIW, back in my PHP days, I did use persistent connections but it would be interesting to see how challenging those are to implement. It may totally be worth the effort.

As far as page caching goes, I do intend for Caddy to get a cache middleware at some point that should be able to take care of caching pages (although dynamic pages are trickier to figure out).

@mattfarina Glad you could join our discussion. 🎉 I hope that by Go 1.6, the http2 lib we're using will be able to handle server push (or have some hooks for it), but at this time I don't think it does.

About the two ways to use PHP, I think this Lua engine can handle both. We can parse Lua out of HTML and it (through the use of require or dofile()) can also execute Lua-only scripts. However I think there needs to be a way to handle the case where a Lua-only script is requested directly... it'd be nice if we could auto-detect whether the file is pure Lua or if it's HTML (or plaintext or anything else) with Lua embedded and handle it appropriately.

[technosophos]

So... Luabrary was originally a typo. I was trying to type Lua Library... but it looked cute, so I left it.

Cookies are a great idea, and I think we can easily accommodate serving from object storage and other things. I think both @mholt and I have delved deeply into the Go HTTP2 library, and it's really not yet at the point where it can meaningfully handle the more advanced HTTP/2 stuff. If I recall correctly, we can get it to do data framing (the HTTP/2 version of chunking). So that one is a great idea to include.

[CMCDragonkai]

Is it possible to make use of this: http://leafo.net/lapis/ ?

[solisoft]

Do you plan to use LuaJIT ? I'm playing with the Lapis Lua framework lately and it's just very good !
What about Lua coroutines ?