本文将简单探讨和比较几种常见的缓存更新策略,以及它们各自的优缺点。
Note: 启发自 《从程序员到架构师:大数据量 缓存、高并发、微服务 多团队协同等核心场景实战》一书的第四章:读缓存
。
更新缓存涉及到两个基本步骤:更新数据库
和更新缓存
。
尽管这两步听起来简单,但在实际操作中需要考虑多个问题:
这些问题的复杂性导致了多种缓存更新组合方案的产生。
我们假设一下
假设,我们采用Redis作为缓存,MySQL作为数据库。
graph LR A[开始] --> B[更新缓存] B --> C[更新数据库] C --> D[结束]
虽然redis支持事务,但是不支持回滚。因此,这种方案可能导致更新数据库失败后,只能手动回滚redis的操作。
如果在多线程同时更新的时候,会导致不一致的问题,如果更新过程中加锁,虽然可以保证数据一致性,但会大大降低系统的可用性。
因此,这种方案成本较高,不推荐使用。
graph LR A[开始] --> B[删除缓存] B --> C[更新数据库] C --> D[结束]
这种方案避免了更新失败需要回滚缓存的问题,但可能引入缓存与数据库不一致的情况。
gantttitle Race ConditiondateFormat YYYY-MM-DDsection A删除缓存 :2014-01-01, 1d更新数据库 :2014-01-04, 1d读取数据(来自B的旧值) :2014-01-05, 1dsection B更新数据库 :2014-01-02, 1d读取数据&建缓存 :2014-01-03, 1d
如图,A和B几乎都在同一时间更新数据X,A先删除了缓存,但是B在A完成数据库更新前,就完成了缓存重建的工作,最后A读取X的时候就是缓存的B的旧值。
按照惯例,依旧可以通过加锁来解决这个问题,但是这样会大大降低系统的可用性。
这里就涉及了高可用性和一致性取舍的问题。这就需要根据具体业务和需求来讨论了。
graph LR A[开始] --> B[更新数据库] B --> C[更新缓存] C --> D[结束]
如果更新数据库成功,而更新缓存失败,本身问题不大,不需回滚,后期在读取的时候,系统会自动尝试建缓存。
gantttitle Race ConditiondateFormat YYYY-MM-DDsection A更新数据库 :2014-01-01, 1d更新缓存 :2014-01-04, 1dsection B更新数据库 :2014-01-02, 1d更新缓存 :2014-01-03, 1d
这样又悲剧了,A把缓存又改成了旧值。
又来了:+🔒,蛋都碎了,为了更新缓存,给数据库读写加锁,有点杀鸡用牛刀了。
graph LR A[开始] --> B[更新数据库] B --> C[删除缓存] C --> D[结束]
这种方案解决了组合3中的一些问题,尤其是在处理并发更新时。通过删除缓存来强制后续的读操作去数据库中获取最新值,从而保证了数据的一致性。但是又引入了新的问题:这个时候B读取的时候,读到的是旧值
。
gantttitle Race ConditiondateFormat YYYY-MM-DDsection A更新数据库 :2014-01-01, 1d删除缓存 :2014-01-03, 1dsection B读取数据 :2014-01-02, 1d
graph LR A[开始] --> B[删除缓存] B --> C[更新数据库] C --> D[再次删除缓存] D --> E[结束]
这个方案试图通过在更新操作前后都删除缓存来最小化数据不一致的风险。尽管这种方案并不完美(如下图,C会读到B存放在缓存中的旧数据
),但它在减少脏数据读取的概率方面做得更好。
gantttitle Race ConditiondateFormat YYYY-MM-DDsection A删除缓存 :2014-01-01, 1d更新数据库 :2014-01-03, 1d删除缓存 :2014-01-05, 1dsection B读取数据(会重建缓存) :2014-01-02, 1dsection C读取数据(B放的旧数据) :2014-01-04, 1d
选择哪种缓存更新策略取决于具体的应用场景和对一致性、可用性以及系统性能的要求。没有一种方案是完美的,每种方案都有可能读取到脏数据,但是通过合理的设计可以最小化这种风险。
原文引用书作者的建议
任何一个方案都不是完美的,但如果剩下1%的问题需要花好几倍的代价去解决,从技术上来讲得不偿失,这就要求架构师去说服业务方,去平衡技术的成本和收益。
共识机制 | 优点 | 缺点 |
---|---|---|
Leader-based(主从) | - 写操作的延迟较低,因为所有写操作都由单个领导者节点处理。 - 实现相对简单,因为领导者节点负责协调所有的复制操作。 | - 如果领导者节点失败,整个系统的写能力会受到影响,直到选举出新的领导者。 - 可能成为系统的瓶颈,因为所有的写操作都需要通过领导者节点。 |
Consensus-based(基于共识) | - 提供强一致性保证,即使在部分节点故障的情况下也能保持数据一致性。 - 由于所有节点参与共识过程,因此系统的容错性较高。 | - 写操作的延迟可能较高,因为需要多个节点之间的共识。 - 实现和理解的复杂性较高,因为共识算法通常比较复杂。 |
Leaderless( (无领导者) | - 高可用性,因为所有节点都可以接受读写请求,没有单点故障。 - 可以提供高读写吞吐量,因为请求可以在多个节点之间分散。 | - 只能提供最终一致性,而不是即时的强一致性。 - 同步和冲突解决可能比较复杂,需要仔细设计以确保数据一致性。 |
Leader-based的典型服务有Apache ZooKeeper。
Consensus-Based的典型服务有etcd。
Leaderless的典型服务有Cassandra。(Cassandra在某些操作,如轻量级事务中,使用Paxos算法来保证操作的原子性和一致性)
这些复制策略的选择取决于系统的特定需求,例如对一致性、可用性和性能的不同要求。
在实际应用中,需要根据应用场景和业务需求来选择最合适的复制策略。
]]>Currently, many backend programmers use Postman to test API requests. However, Postman as atool itself does not provideparticularly powerful JSON data visualization. For example, it only offers simple code-level data folding and cannotperceive the specific number of elements inside a list.
To compensate for Postman’s shortcomings, I often use JSON Editor Online to parse andvisualize JSON data moreeffectively. The Tree View feature allows us to focus on the data of interest by collapsing irrelevant parts. However,it dampens my enthusiasm as it requires manual copying and pasting of data each time.
But after closely examining Postman, I discovered the existence of a “Visualize” tab. My intuition tells me that thiscould be the solution I need.
In short, let’s jump straight to the demo: By enabling the JSON tree view plugin, we can achieve the following effect.
Thanks to the opne source project: josdejong/jsoneditor
]]>The story is that I have a feature branch, and I keep merging new sub-feature PRs into that main feature branch.
As a milestone, I would like to rebase the main feature branch after a while. So I type the code without any hesitation.
1 | git fetch origin && git rebase origin/main |
I did not realize anything bad, until I open a new PR to merge a new module to the feature branch, and I can see:
OK, I am fine! 😭 (Not a big deal with a force push without a pull?!)
After few seconds with trying ctrl + z
, and nothing happens. I know it is time to take some real actions.
feature/my-awesome-feature
We can see this log info from the pr page, and we will know: before the force push, the head is @ 1234567
. And we needthis number.
If you run git fetch origin && git rebase origin/main
, you may see the similar log below. And you can see the latesthead of feature/my-awesome-feature
is 1234567
.
1 | From github.com:XYZ/ABC |
With the commit hash we get before the force push. We can restore branch feature/my-awesome-feature
as
1 | $ git reset --hard 1234567 |
Recently, I migrated the SaltyNote service from MariaDB to MongoDB. While the reason is not for high throughput orperformance.
Be host, the important reason I changed to Mongo is that I need a more flexible schema for the note data.
If I stick to MariaDB, I need to have a separate table for note tags field for better aggregation and search later.
Info
I know some SQL database support JSON field, that can also be the solution. While instead of using JSON field, I prefer NoSQL.
1 | create table note ( |
Without JSON field, I need to create a separate table for note tags field, so I can query note by tags easily.For example, to query all notes with tag java
, I can use the following SQL:
1 | # Using * here is not a good practice, just for demo |
While this solution will bring some trouble when update tags, for example, if I have a note with tags java
and spring
.Later, I need to update the tags to java
and spring-boot
. I need to delete the old tags and insert the new tags. Thefull update will include multiple actions.
1 | delete * from tag where note_id = 1; |
1 | create table note ( |
With JSON field, we can store the tags in the same table. And the update will be much easier, since we do not need toscan multiple rows.
1 | # query all notes with tag `java` |
On my research, I find Spring-Data-JPA does not support JSON field well. So I need to use native SQL to query andupdate, or introduce new dependency: HibernateTypes.
For the reason above, eventually, I choose MongoDB. With MongoDB, I can store the tags in the same document with thenote object.
Info
Transaction is not a concern for SaltyNote. Since it is OK to save duplicated notes, and user can simply delete it. (No duplicated notes have been found so far)
The tag feature will be available in the next releaseof SaltyNote Chrome Extensionv0.4.0.
]]>Reference: https://saltynote.github.io/saltynote-chrome-extension/
Unlike standard website, Chrome Extension can have 2 different runtime environments: background script and contentscript.
Same-Origin Policy
, the content script cannot send http requests to the SaltyNote Server directly. So thecontent script will send a message to the background script, and the background script will forward the http requestto the SaltyNote Server. Once the response is received, the background script will send the response back to thecontentscript. When injecting the content script, a css file will also be pushed to host page. If we do not take if carefully, theinjected css may impact the UI of the host page. For resolve this, Iuse sass-plus
(Source Code)to generate a customized css filewith bootstrap within a custom classname saltynote
(source code).Then I will add the class name to the root element of the Vueapplication. So the css will only impact the elements within the Vue application.
In this article, I will introduce the implementation of the SaltyNote service.
The Code base: https://github.com/SaltyNote/saltynote-service
User
, Note
, Vault
, Login History
. While, I may migrate it to NoSQL as mongodb in the future.When I was laid off, I did not take it very serious, since I was still in paternity leave, and I spent some a lot oftime with my kid, and spare some time for interview preparing.I was pretty confident that I would find my new job if I want to even I will be paid less.
But later, I find it is not that easy.I have been interviewed by some companies, and I have been rejected by all of them eventually. I do not think I did abad job, maybe there are better candidates than me, or maybe I am not a good fit for the company.
For example, a startup uses Golang for backend, and I told the recruiter that I have very little experience with it. Ieven spent a full day for learning Golang, so that I can use golang for interview. I did not do any wrong, and I ampretty happy with my performance. But I was rejected anyway.
Even for a senior software engineer as me, it is not easy to find a new job. I can feel it will be super hard year fornew graduates.
Best of luck for all of us!
]]>Login database from terminal with mysql -u root -p
, and execute follow command lines:
1 | CREATE DATABASE saltynote; |
Note: You can find more details about how to enable remote access from this post. While here, I only enable it with local access, as I for security reason, it is not required for current stage.
The service is implemented with Spring Boot. So it can be run with a standalone jar file as a Systemd service.You can find more official information from this link.
For security, I will create a specific user to run the service. This post will be very helpful for this step.
1 | # Create a new user, and create its home dir |
1 | mkdir -p /home/saltynote/service |
Upload the jar file to /home/saltynote/service
folder, and create application.properties
inside that folder, which can be used to set some sensitive information. e.g. database connection info.
1 | # Make saltynote is the owner of service.jar |
Create note.service
in /etc/systemd/system
dir, and populate note.service
as below:
1 | [Unit] |
We can enable this service to auto start when system restarts by:
1 | systemctl enable note.service |
If everything goes well, the service should start now.Open http://YOUR-SERVER-IP:8888, you should see a welcome message in JSON format.
1 | # You can check that the service is running with saltynote user. |
Now, we can connect NginX with our service, which means we want http://dev.saltynote.com
displays the same contentas http://dev.saltynote.com:8888
instead of showing NginX welcome page.
Open /etc/nginx/nginx.conf
, and add a new upstream
inside http
section. (I named it as service
, while it can beany name you want.)
1 | http { |
Open /etc/nginx/sites-available/default
, and update the server_name
and location
sections.
Note: You must own the domain before you can add them to server_name
.
1 | server { |
1 | service nginx restart |
So far, everything seems working now, while the browser still complains Not Secure
in the url bar. It is time toenable https for our service. It is free with Let’s Encrypt.
It should be easy to install certbot by followingthe instruction.
1 | # Since I use nginx, I will run this to enable https |
We enable NginX with https for our service now, and we can hide the original service running in port 8888
from thepublic. There are multiple solutions for this. e.g. iptables
.
While I find UFW - Uncomplicated Firewall
is more user-friendly, so I willchoose it here.
1 | ➜ ~ ufw status |
Note:For enhanced security, you can change default SSH port 22 to other number.
Now you will not be able to access the service with https://dev.saltynote.com.
The step 1 is what I will do for all new servers, even for my personal Linux laptop.
1 | # This step may take some time, just be patient |
When this step is done, you can restart your server. But usually, I will restart after I have everything installed.
1 | # Maybe most of them are already bundled with Ubuntu |
I really like zsh
, and I will also enable oh-my-zsh. But this is optional, even I highly recommend it.
1 | sudo apt install openjdk-11-jre mariadb-server nginx |
I install open Java 11 here, based on your own requirement, you can either install Java 8 or Java 14, whatever you want.The reason why I just install JRE not JDK is because I will only need Java Runtime in this server, and I will never try to compile or build any Java projects.
1 | # Check whether java is installed correctly |
It is almost the same as MySQL. Personally, I choose Mariadb these days.An important reason is that they have different licenses.
If you take Open Source seriously, this is something that you should care. 如何选择开源许可证
Configure MariaDB
1 | sudo mysql_secure_installation |
Verify MariaDB
1 | # You should see similar output as below: |
As a reverse proxy, NginX is very helpful. Usually, I use it as a load balancer.
Simply open http://your-server-ip
, you will see a page showing these words
1 | Welcome to nginx! |
You can download the stylish here, and import it to stylus.
BTW: I have some some hard-coded style for my 2560 x 1440(or similar) screen. I also fix the style for github home page.
Another improve for github - add personal&star links in head menu This is a tampermonkey script, which can be download here. (This is also a modified version from existing one, thanks for the good work from open source contributors)
]]>/usr/share/applications/netease-cloud-music.desktop
添加一个 --no-sandbox
就可以解决了。1 | Exec=netease-cloud-music --no-sandbox %U |
但是等我升级之后,网易云音乐怎么也启动不了。后来在知乎中再次看到了解决方法,顺便在此记录一下。
1 | # 需要在网上搜索并下载网易云音乐1.0.0版本.deb |
如果你嫌麻烦,你可以直接下载我修改好的了,netease-cloud-music_1.0.0_amd64_modified.deb. 如果你不放心,可以自己打包,也不是很复杂。
Note: 在这个方式中你不加--no-sandbox
也没有问题。
点击unblock-NetEaseMusic获取更多信息。
]]>Searched and tried a lot, and finally, I find the correct one
(which works for me).The link is: https://ubuntuforums.org/showthread.php?t=2340769
1 | # Uninstall rtl8812au-dkms if you installed it as other solutions told you |
1 | jackson InvalidDefinitionException: |
1 | // assume there are many teams of students in a class, and each group has a team leader |
For serialization, there will not be any problem, and we can have the print-out
1 | {"teamMap":{"Student{name='Salty Egg', age=20}":[{"name":"Sweet Egg","age":20}]}} |
We can find that when serializing the map key, Jackson
will call toString()
method by default.
fix
thisThe standard solution is to write the key serialize/deserialize
for Jackson
, what if I do not wanna it as this?!
1 | public class KeyValueContainer<K, V> { |
Then, with the same test case above, we can have the following result:
1 | {"team":[{"key":{"name":"Salty Egg","age":20},"value":[{"name":"Sweet Egg","age":20}]}]} |
Note:
getter & setter
as private
, the third-party developers will not call it with mistakesList
, which may bring a bit performance issue.Note: What I want to do are:
Let’s see a quick demo first: (Book
has its Author
, while Author
keeps a list of his/her Book
s)
1 | public class Book { |
Now, we can have a serialization sample as:
1 | public static void main(String[] args) throws IOException { |
Then it will throw this exception:
Exception in thread "main" com.fasterxml.jackson.databind.JsonMappingException: Infinite recursion (StackOverflowError) (through reference chain: ...
JSON is pretty hard to demonstrate the data with Circular References
Thx to Jackson, with @JsonIdentityInfo, we can easy fix this issue.After I add @JsonIdentityInfo(generator=ObjectIdGenerators.IntSequenceGenerator.class, property="@id")
to both class, then I can have the JSON string as:
1 | {"@id":1,"name":"Book","author":{"@id":2,"name":"Author","books":[1]}} |
1 | {"@id":1,"name":"Author","books":[{"@id":2,"name":"Book","author":1}]} |
However, I am not sure whether this JSON string can be deserialized by other libraries(e.g. GSON), or other languages, my guess is NOT.
Note: Think twice before using it!
]]>一阵Google之后,找到了这个https://intellij-support.jetbrains.com/hc/en-us/community/posts/115000160850--resolved-JetBrains-Toolbox-Error-creating-SSL-context里面提到在新版本中会修复这个问题,问题是我不能就这么坐以待毙等下一个版本吧。
于是在(这个里面)[https://youtrack.jetbrains.com/issue/ALL-1486]找到了临时的解决方法。
错误原因就是toolbox不支持openssl1.1+版本。
sudo
)1 | mkdir -p "/opt/openssl-1.0.2k" |
1 |
|
Note: 如果设置了toolbox的开机启动,就会发现还是会有链接问题,只需要关了,再通过启动脚本打开就行。坐等JB公司更新toolbox新版本。
]]>也试过用其他的在线API管理工具,太懒了,总是无法保持更新。而且大多数服务还是收费的。
最近一次,我在试用EOApi,终于被它的访问速度打败了(貌似这个对path parameter支持的也不是很好),我不得不寻找新的工具。
至于为什么最后会选择Swagger,我已经记不清了。其实对于Swagger我一直是有耳闻的,只是一开始我玩弄Swagger Editor的时候,我以为它只能从YAML/JSON生成Java代码的呢。没想到居然反着也行。
长话短说,在此就简单介绍一下如果做Swagger, Spring-Boot和Jersey的快速集成。(其实我是根据这个链接来学习的: 点我)
Spring-Boot
的项目,添加必要的依赖1 | <dependency> |
Jersey
和Swagger
Note: Swagger不和Jersey共用一个Jackson的MapperObject
, 所以你如果要让你的Model/Entity显示为SNAKE_CASE
,你需要加上
1 | io.swagger.util.Json.mapper().setPropertyNamingStrategy(PropertyNamingStrategy.SNAKE_CASE); |
但是这也会导致一个问题,如果你以http://127.0.0.1:8080/swagger.json
来获取内容,你会发现那些basePath
和operationId
也变成了SNAKE_CASE
,Swagger-UI会报错。其实这个时候你只要换成http://127.0.0.1:8080/swagger.yaml
一切就妥妥的了。
Note: 在Spring-boot
中,所有的Jesey的EndPoint必须一个个添加,不能直接package(XXX)
之类的,Spring-boot
官方文档有提到,我之前踩过这个坑。
1 |
|
HelloWorld resource
Note: @SwaggerDefinition
这里面不是必要的,我加在这里只是为了提示Swagger有security方面的注解
1 |
|
Swagger-UI
来显示我们的API文档其实我还没有整明白如果实现http-header
的basic authorization
。等我哪天知道了再过来更新。
I go through hexo plugin page very carefully, and find there are 2 related plugins may be helpful for me:
I tried to use the latter one(I like JSON more than XML), but find I cannot use it well, as I am not a JS expert. Also, I googled some Open-Source search engine tools, and I finally find TIPUE SEARCH
Now I have the Open-Source search engine tool and JSON generator. But the JSON generated by hexo-generator-json-content
does not fit Tipue Search
. Comes 2 solutions now:
hexo-generator-json-content
Tipue Search
This is not a difficult decision, and I prefer the first one.
So I create this repo: Hexo-Tipue-Search-Json
You can find all information you want here: https://github.com/zhouhao/Hexo-Tipue-Search-Json
先是助理医生让我坐在椅子上,不由自主的深呼吸了一下,因为我也不知道接下来究竟是不是很可怕。然后助理医生搬来一堆子器械,看到那些钳子,剪子之类的,内心开始慢慢动摇了—-难道非要拔牙不可吗?没过一会,助理医生过来用棉签占了一些红色的东西在我的智齿附近涂抹,并塞了一些纱布进我口腔,当时只是感觉有点反胃,然后牙龈就开始火辣辣的了。
终于拔牙医生过来,很客气的说”How are you?”, 我都不好意思回答”I am fine”, 默默说了一句”I am a bit nervous”,认怂了。
借助之前棉签的作用,打麻醉的时候反到没拿疼,但是还是能感觉医生在一块区域不停的把麻醉针扎来扎去。
麻醉过后大约10到15分钟医生就准备给我拔牙了,那阵势真是有点可怕,大钳子直接伸进嘴里。第一颗(右上)很顺利,我都没感觉到就下来了,当时就想要是都这样就好了,但是我的运气就从没那么好过。第二颗(左上)的时候,我听见牙齿裂掉的声音了,日,医生和助理医生在那个牙龈处用“铲子”刮来刮去,反复拔,最后我也不知道是不是干净了,当时我一心就想快点结束。
下面两颗貌似比较麻烦,他们居然先用电动的矬子磨我牙,然后用一根铁杆借助杠杆原理往上扳我牙。他奶奶的,我当时居然感觉到痛了,无奈医生就又补了几针麻醉,然后准备先拔掉左下。也不是那么干净利落,但也是拔下来了。他们还给我缝了几针。最后一个牙齿后来还是疼,接着补了麻醉,然后终于拔下了。拔下的那一瞬间,心情顿时就舒缓了。
口腔塞着纱布,好奇怪,一开始还能支支吾吾说几句,但是当我从CVS买完药的时候发现舌头终于撸不直了,害得我问个厕所都得打字给别人看。
最后就去理了个头,算是弥补一下我的形象!负分~
]]>首先,我得承认为什么拖了一年才看牙医,纯粹是因为我懒。最近正好烦恼太多,无法安静干些“大”事,于是就准备处理牙。
之前问过同事,同事也给了建议,后来一拖全忘了,还好有好心人在公司内部收集了一个牙医推荐。我选了Seligman,但是没有处于任何原因。但是整个过程除了疼,还是很好的。
没什么好多的,美国凡事都得预约。又得用一下蹩脚的英语
一个医生用一个含着很不舒服的东西(有棱角,有些角度还是有点疼的),不停的换位置,让我夹着,给牙照相。拍完,来个一个看上去专业点的医生指着牙的部位对我说,左侧最里面智齿蛀掉一半,补牙成本太高,建议我拔,右边也是,虽然轻点,但是也要拔。为了上下平衡,把上面两颗对应的也要拔掉,于是就要拔掉4颗牙。为了安慰我,他说他和那个检查的医生都把那些牙拔了。WTF好吧,按照美国办事的尿性,肯定不会当天拔牙,于是医生给了我另一个人的信息,说是专门做牙科手术的,让我去预约时间。
当然,他们还顺便推荐我当天洗牙
妈蛋,我以为就是简单洗洗,居然疼死我了。他们用不知道什么样的设备在我牙上刮来刮去(医生说全程闭眼会舒服点),几乎刮破了所有能刮破的牙龈(牙齿和肉衔接的地方)。漱口的时候全是血。因为洗牙需要用水,他们就在我口腔放了一个吸气的管子,问题是很多时候它吸到我肉上了,唉,肉疼。洗完之后,所有的医生都问我“舒服吗?”,舒服你妹啊,疼死老子了。
于是悲剧的一天几乎就在满嘴血腥味中要结束了。
]]>