Can I create a custom module that works like Quotes with Line Items?

No, Vtiger does not allow adding a Line Item block to a custom module. Line item functionality, where you can add products/services, quantities, prices, and taxes, is available only in standard inventory modules such as Quotes, Invoices, Sales Orders, and Purchase Orders.