Technical Bi-Weekly Summary (Aug 31th)
This is the first weekly technical summary. These summaries will review the team's technical accomplishments. We'll cover various components, but may not delve as deeply as some topics warrant, given that many could merit individual blog posts. For the sake of balance, we'll strive to offer a comprehensive overview. We aim to produce high-quality summaries as time and bandwidth allow, with the goal of making this a regular update.
General Updates
The Shinkai team has recently released two important pieces of content. The first is a video showcasing the utilization of vector resources in Shinkai, providing insight into how these resources are being utilized by the team. Additionally, they have published a blog post that delves into the overlay OS for personal AI and highlights the unique approach taken by Shinkai compared to traditional operating systems. The community has shown us a lot of positive support with this new content, and we're excited to share more in the coming weeks.
Shinkai Tools and Toolkit Integration
Overview:
Shinkai is a Personal AI OS which seamlessly manages your personal data together with powerful tools that unlock new capabilities for your large language model (LLMs) based AI agents. This update focuses on the tool and toolkit integration aspect of Shinkai.
Key Concepts:
- Vector Resources: A foundational component for storing and managing data. Two main types are Document and Map Vector Resources.
- Document Resources: Store ordered chunks of data, ideal for PDFs and web content.
- Map Vector Resources: Newly introduced to handle key-value pairs, designed for more flexible data manipulation.
- Toolkits & Tools: Extend the capabilities of Shinkai by adding functionalities via third-party toolkits, which contain individual tools.
- Tool Router: An indexing system that makes tools from installed toolkits searchable and accessible for LLMs.
Major Updates:
- New Vector Resource Type: The Map Vector Resource has been added to handle toolkits and tools. It uses a hash map as its internal data model to facilitate quick data manipulation.
- Tool Router Implementation: Manages all installed tools, making them easily searchable by LLMs. Two types of tools (Rust and JavaScript) have been accommodated.
- Database & Installation Flow: In development, focusing on backend processes to install, validate, and activate toolkits and tools.
- Persistence Features: Features like header persistence have been introduced to make the user experience more seamless.
Upcoming Work
- Finalize the database integration for toolkit installation and activation.
- Implement a robust header validation mechanism.
- Complete the execution path for Rust tools.
Node.js Execution Engine Development
Overview:
- The team has focused on enhancing the execution engine for Node.js TS tools, aiming for consistent error outputs, security, and performance optimization.
Key Concepts:
- Executor: Manages the execution of Node.js code.
- Single-threaded Process: Utilizes a single process for execution to streamline operations.
- Worker Threads: Employs Node's worker threads for efficient, lightweight execution.
- Security: Implemented features to isolate memory spaces between worker threads and the main process, reducing security risks.
Major Updates:
- Executor Optimization: Transitioned to a single-threaded, single-process execution model that doesn't rely on external files.
- Error Handling: The system is now robust enough to capture a wide array of errors, enhancing its reliability.
- Memory Management: The system has demonstrated stability under various load conditions. Solutions are in place to manage RAM efficiently, especially for smaller machines.
Upcoming Work:
- Task Queuing: A queuing system is in the pipeline to enhance task management.
- Security Features: Additional measures will be added to further restrict unauthorized file system and network access.
Tools and Development:
- Linting Rules: A setup is under development to alert developers against using restricted libraries, thereby ensuring code quality and security.
This framework outlines the significant updates and forward-looking plans for the project. The team is actively working to implement these features.
Development of Shinkai Visor
Overview:
- A new hire will work on developing Shinkai Visor and support the development of the Shinkai App. The Shinkai App focuses on admin capabilities and basic conversation UI, while Shinkai Visor serves as a user-facing Chrome extension for data extraction and management.
Key Concepts:
- Shinkai App: Admin dashboard for managing the Shinkai node.
- Basic Conversation UI: This will facilitate user interaction within the Shinkai environment.
- Shinkai Visor: A Chrome extension modeled after Urbit Visor.
- Data Extraction: Extracts webpage content and saves it into the database as vector resources.
Major Updates:
- Shinkai Visor Capabilities: The extension will allow users to scrape content from different web pages and save them in various formats.
- Data Ingestion: Shinkai Visor will support both PDFs and on-page HTML content.
- Custom Extensions: Potential for specialized plugins for specific websites or data types, like Twitter timelines or Google Sheets data.
Upcoming Work:
- Data Parsing: Implementation of default parsers for various file formats such as PDF, EPUB, CSV, and DOCX.
- User Interface and Data Flows: Exploration of how data is ingested via user interaction and the corresponding impact on data extraction needs.
Tools and Development:
- Web Scraping: Specialized extensions that interface with Shinkai Visor for data extraction from specific web pages.
- Vector Resources: Introducing different types of vector resources like documents and hash maps for storing scraped data.
The project aims to create a robust system for data extraction and management via Shinkai App and Shinkai Visor. Plans are in place to extend capabilities based on user needs.
Shinkai Node and Shinkai App
Overview:
- Update on the state of various components within the Shinkai ecosystem, focusing on state management, agent integration, and WASM library updates.
Key Concepts:
- State Management: Updated to use enums like "encrypted" or "unencrypted" instead of flags. This was also updated for the WASM and Typescript implementations.
- Agent Integration: Integration of agents like ChatGPT 3.5 Turbo into the system. Agents can perform tasks or execute jobs.
Major Updates:
- WASM Library Updates: The library now includes all core components, including "Shinkai Name" and "Inbox Name".
- TypeScript Wrappers: These have been stabilized and made easier to use.
- Endpoint Extensions: New endpoints are being added to the Shinkai App for features like agent listing and permissions.
Upcoming Work:
- Database Capabilities: Implementation to save Twitter timeline data and other web resources into Shinkai nodes.
- HTTP Interface for Primitives: Discussion on creating a lightweight system for executing tasks directly at the node level without using agents.
Issues & Considerations:
- Agent & Job Model: Hesitation to extend the job model for lightweight tasks, proposing the idea of creating a new system just for the execution of primitives.
- Vector Resources: Currently only accessible via AI agents and tools. Plans to make these accessible directly to external clients.
Discussion Points:
- Permissions & Security: How to safely expose Rust tools or capabilities to external devices without requiring an agent.
- Unified Interface: The need for a unified system to handle both internal and external vector resources.
- State vs Enum: The transition from flag-based states to enum-based states and its ramifications across the repo.
The team is progressing on multiple fronts, balancing between updating existing systems and integrating new features. The discussion brings forth questions on architecture and the need for streamlined ways to interact with the Shinkai node.
Identity Management with ERC-721 NFTs and Metadata Server
Overview:
- The team has been working on redefining the ownership of identity within a blockchain system, making it correspond to an ERC-721 NFT. An associated metadata server and event system are also in development to support this feature.
Key Concepts:
- Identity Ownership: Transformed from address-based to ERC-721 NFT-based.
- Metadata Server: This will serve the required metadata for each identity NFT.
Major Updates:
- Identity-NFT Mapping: Successfully mapped ERC-721 NFTs to identities and implemented minting and burning operations.
- Event Refactoring: Optimized the event system for lower gas costs and less complexity.
- On-Demand Info Retrieval: Shifted from full indexer requirement to on-demand information querying using RPC endpoints.
Upcoming Work:
- Event Documentation: Plan to document the events for potential future indexer needs.
- Test Updates: Extending the range of testing for new features and refactors.
Tools and Development:
- RPC Provider: Information retrieval will leverage RPC rather than indexers.
- Caching Strategy: Contemplating Time-To-Live (TTL) caching initially, followed by more advanced post-caching mechanisms.
The project is steering towards a minimalistic yet efficient architecture that doesn't rely heavily on full indexers but uses on-demand RPC calls to get current blockchain states. Plans for future improvements, including caching strategies and event documentation, are also under consideration.
