[==========] Running 57 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 31 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 03 08:07:40 dist-test-slave-27r9 krb5kdc[13102](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 03 08:07:40 dist-test-slave-27r9 krb5kdc[13102](info): set up 2 sockets
Sep 03 08:07:40 dist-test-slave-27r9 krb5kdc[13102](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20250903 08:07:42.239534 13095 mini_kdc.cc:121] Time spent starting KDC: real 1.968s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/krb5kdc/test-user.keytab.
Sep 03 08:07:42 dist-test-slave-27r9 krb5kdc[13102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886862, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-09-03T08:07:42Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-03T08:07:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.201.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.201.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.201.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.201.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.201.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.201.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.201.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.201.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/krb5kdc/kudu.keytab.
I20250903 08:07:42.370421 13095 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUML90r/build/debug/bin/kudu
/tmp/dist-test-taskUML90r/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.201.254:36733
--webserver_interface=127.12.201.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.201.212:43327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.201.254:36733
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 08:07:42.475481 13118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:07:42.475731 13118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:07:42.475786 13118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:07:42.479002 13118 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250903 08:07:42.479068 13118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 08:07:42.479091 13118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:07:42.479110 13118 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250903 08:07:42.479128 13118 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250903 08:07:42.483359 13118 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.201.212:43327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.201.254:36733
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.201.254:36733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.201.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 07:50:08 UTC on 5fd53c4cbb9d
build id 7874
I20250903 08:07:42.484498 13118 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:07:42.485390 13118 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:07:42.491014 13123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:07:42.491014 13126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:07:42.491117 13124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:07:42.491349 13118 server_base.cc:1047] running on GCE node
I20250903 08:07:42.491815 13118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:07:42.492686 13118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:07:42.493830 13118 hybrid_clock.cc:648] HybridClock initialized: now 1756886862493821 us; error 26 us; skew 500 ppm
Sep 03 08:07:42 dist-test-slave-27r9 krb5kdc[13102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886862, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 08:07:42.496304 13118 init.cc:377] Logged in from keytab as kudu/127.12.201.254@KRBTEST.COM (short username kudu)
I20250903 08:07:42.497326 13118 webserver.cc:480] Webserver started at http://127.12.201.254:35975/ using document root <none> and password file <none>
I20250903 08:07:42.497867 13118 fs_manager.cc:362] Metadata directory not provided
I20250903 08:07:42.497920 13118 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:07:42.498101 13118 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:07:42.499858 13118 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/minicluster-data/master-0/data/instance:
uuid: "88cd83afbe7a4ceebac3208618002642"
format_stamp: "Formatted at 2025-09-03 08:07:42 on dist-test-slave-27r9"
server_key: "d20757ab451370efea4e7b3aa5006900"
server_key_iv: "b4e1fd035c632914b7c9db9e0dfdeb42"
server_key_version: "encryptionkey@0"
I20250903 08:07:42.500329 13118 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/minicluster-data/master-0/wal/instance:
uuid: "88cd83afbe7a4ceebac3208618002642"
format_stamp: "Formatted at 2025-09-03 08:07:42 on dist-test-slave-27r9"
server_key: "d20757ab451370efea4e7b3aa5006900"
server_key_iv: "b4e1fd035c632914b7c9db9e0dfdeb42"
server_key_version: "encryptionkey@0"
I20250903 08:07:42.503796 13118 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250903 08:07:42.506006 13133 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:07:42.506958 13118 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250903 08:07:42.507113 13118 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/minicluster-data/master-0/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/minicluster-data/master-0/wal
uuid: "88cd83afbe7a4ceebac3208618002642"
format_stamp: "Formatted at 2025-09-03 08:07:42 on dist-test-slave-27r9"
server_key: "d20757ab451370efea4e7b3aa5006900"
server_key_iv: "b4e1fd035c632914b7c9db9e0dfdeb42"
server_key_version: "encryptionkey@0"
I20250903 08:07:42.507207 13118 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:07:42.521670 13118 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:07:42.524452 13118 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:07:42.524639 13118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:07:42.531545 13118 rpc_server.cc:307] RPC server started. Bound to: 127.12.201.254:36733
I20250903 08:07:42.531543 13185 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.201.254:36733 every 8 connection(s)
I20250903 08:07:42.532522 13118 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/minicluster-data/master-0/data/info.pb
I20250903 08:07:42.534963 13186 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:07:42.536032 13095 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUML90r/build/debug/bin/kudu as pid 13118
I20250903 08:07:42.536164 13095 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/minicluster-data/master-0/wal/instance
I20250903 08:07:42.537091 13095 external_mini_cluster.cc:1468] Setting key f82d7d816f395ac5c06451108f2a432a
I20250903 08:07:42.540656 13186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88cd83afbe7a4ceebac3208618002642: Bootstrap starting.
I20250903 08:07:42.543159 13186 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88cd83afbe7a4ceebac3208618002642: Neither blocks nor log segments found. Creating new log.
I20250903 08:07:42.543991 13186 log.cc:826] T 00000000000000000000000000000000 P 88cd83afbe7a4ceebac3208618002642: Log is configured to *not* fsync() on all Append() calls
I20250903 08:07:42.545826 13186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88cd83afbe7a4ceebac3208618002642: No bootstrap required, opened a new log
I20250903 08:07:42.548900 13186 raft_consensus.cc:357] T 00000000000000000000000000000000 P 88cd83afbe7a4ceebac3208618002642 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88cd83afbe7a4ceebac3208618002642" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 36733 } }
I20250903 08:07:42.549149 13186 raft_consensus.cc:383] T 00000000000000000000000000000000 P 88cd83afbe7a4ceebac3208618002642 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:07:42.549208 13186 raft_consensus.cc:738] T 00000000000000000000000000000000 P 88cd83afbe7a4ceebac3208618002642 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88cd83afbe7a4ceebac3208618002642, State: Initialized, Role: FOLLOWER
I20250903 08:07:42.549671 13186 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88cd83afbe7a4ceebac3208618002642 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88cd83afbe7a4ceebac3208618002642" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 36733 } }
I20250903 08:07:42.549790 13186 raft_consensus.cc:397] T 00000000000000000000000000000000 P 88cd83afbe7a4ceebac3208618002642 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250903 08:07:42.549856 13186 raft_consensus.cc:491] T 00000000000000000000000000000000 P 88cd83afbe7a4ceebac3208618002642 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250903 08:07:42.549960 13186 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 88cd83afbe7a4ceebac3208618002642 [term 0 FOLLOWER]: Advancing to term 1
Sep 03 08:07:42 dist-test-slave-27r9 krb5kdc[13102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886862, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.201.254@KRBTEST.COM
I20250903 08:07:42.550755 13186 raft_consensus.cc:513] T 00000000000000000000000000000000 P 88cd83afbe7a4ceebac3208618002642 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88cd83afbe7a4ceebac3208618002642" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 36733 } }
I20250903 08:07:42.551082 13186 leader_election.cc:304] T 00000000000000000000000000000000 P 88cd83afbe7a4ceebac3208618002642 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88cd83afbe7a4ceebac3208618002642; no voters:
I20250903 08:07:42.551370 13186 leader_election.cc:290] T 00000000000000000000000000000000 P 88cd83afbe7a4ceebac3208618002642 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250903 08:07:42.551677 13192 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 88cd83afbe7a4ceebac3208618002642 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 08:07:42.551858 13192 raft_consensus.cc:695] T 00000000000000000000000000000000 P 88cd83afbe7a4ceebac3208618002642 [term 1 LEADER]: Becoming Leader. State: Replica: 88cd83afbe7a4ceebac3208618002642, State: Running, Role: LEADER
I20250903 08:07:42.552147 13192 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88cd83afbe7a4ceebac3208618002642 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88cd83afbe7a4ceebac3208618002642" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 36733 } }
I20250903 08:07:42.552477 13186 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88cd83afbe7a4ceebac3208618002642 [sys.catalog]: configured and running, proceeding with master startup.
I20250903 08:07:42.553231 13192 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88cd83afbe7a4ceebac3208618002642 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88cd83afbe7a4ceebac3208618002642. Latest consensus state: current_term: 1 leader_uuid: "88cd83afbe7a4ceebac3208618002642" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88cd83afbe7a4ceebac3208618002642" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 36733 } } }
I20250903 08:07:42.553435 13192 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88cd83afbe7a4ceebac3208618002642 [sys.catalog]: This master's current role is: LEADER
I20250903 08:07:42.553838 13200 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250903 08:07:42.555184 13191 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88cd83afbe7a4ceebac3208618002642 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "88cd83afbe7a4ceebac3208618002642" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88cd83afbe7a4ceebac3208618002642" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 36733 } } }
I20250903 08:07:42.555320 13191 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88cd83afbe7a4ceebac3208618002642 [sys.catalog]: This master's current role is: LEADER
I20250903 08:07:42.556859 13189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:42.539307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53688 (local address 127.12.201.254:36733)
0903 08:07:42.539975 (+ 668us) server_negotiation.cc:207] Beginning negotiation
0903 08:07:42.539985 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0903 08:07:42.540124 (+ 139us) server_negotiation.cc:408] Connection header received
0903 08:07:42.542871 (+ 2747us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:07:42.542885 (+ 14us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:07:42.543183 (+ 298us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:07:42.543450 (+ 267us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:07:42.545430 (+ 1980us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:42.546190 (+ 760us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:07:42.547026 (+ 836us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:42.547207 (+ 181us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:42.550760 (+ 3553us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:07:42.550780 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:07:42.550793 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:07:42.550832 (+ 39us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:07:42.553129 (+ 2297us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:07:42.553713 (+ 584us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:07:42.553721 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:07:42.553729 (+ 8us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:07:42.553822 (+ 93us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:07:42.554139 (+ 317us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:07:42.554142 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:07:42.554144 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:07:42.554512 (+ 368us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:07:42.554704 (+ 192us) server_negotiation.cc:1036] Waiting for connection context
0903 08:07:42.555922 (+ 1218us) server_negotiation.cc:300] Negotiation successful
0903 08:07:42.556158 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":114,"threads_started":1}
I20250903 08:07:42.556938 13200 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250903 08:07:42.561918 13200 catalog_manager.cc:1349] Generated new cluster ID: 3b88cc39ba0244b7b4d12b3b0357131a
I20250903 08:07:42.561983 13200 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250903 08:07:42.577400 13200 catalog_manager.cc:1372] Generated new certificate authority record
I20250903 08:07:42.578433 13200 catalog_manager.cc:1506] Loading token signing keys...
I20250903 08:07:42.599304 13200 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 88cd83afbe7a4ceebac3208618002642: Generated new TSK 0
I20250903 08:07:42.600049 13200 catalog_manager.cc:1516] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.201.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.201.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.201.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.201.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.201.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.201.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.201.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.201.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/krb5kdc/kudu.keytab.
I20250903 08:07:42.655862 13095 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUML90r/build/debug/bin/kudu
/tmp/dist-test-taskUML90r/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.201.193:0
--local_ip_for_outbound_sockets=127.12.201.193
--webserver_interface=127.12.201.193
--webserver_port=0
--tserver_master_addrs=127.12.201.254:36733
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.201.212:43327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 08:07:42.782812 13214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:07:42.783046 13214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:07:42.783133 13214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:07:42.786267 13214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 08:07:42.786326 13214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:07:42.786448 13214 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.201.193
I20250903 08:07:42.790640 13214 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.201.212:43327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.201.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.201.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.12.201.254:36733
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13214
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.201.193
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 07:50:08 UTC on 5fd53c4cbb9d
build id 7874
I20250903 08:07:42.791778 13214 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:07:42.792629 13214 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:07:42.799535 13222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:07:42.799535 13220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:07:42.799535 13219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:07:42.799669 13214 server_base.cc:1047] running on GCE node
I20250903 08:07:42.800207 13214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:07:42.800777 13214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:07:42.801966 13214 hybrid_clock.cc:648] HybridClock initialized: now 1756886862801934 us; error 40 us; skew 500 ppm
Sep 03 08:07:42 dist-test-slave-27r9 krb5kdc[13102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886862, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 08:07:42.804855 13214 init.cc:377] Logged in from keytab as kudu/127.12.201.193@KRBTEST.COM (short username kudu)
I20250903 08:07:42.805907 13214 webserver.cc:480] Webserver started at http://127.12.201.193:45253/ using document root <none> and password file <none>
I20250903 08:07:42.806458 13214 fs_manager.cc:362] Metadata directory not provided
I20250903 08:07:42.806514 13214 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:07:42.806705 13214 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:07:42.808488 13214 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/minicluster-data/ts-0/data/instance:
uuid: "ef1366fcc0cf46e9b90f47de268473a9"
format_stamp: "Formatted at 2025-09-03 08:07:42 on dist-test-slave-27r9"
server_key: "66c4c6e84f321deb00b5e56ee231e5ca"
server_key_iv: "02877f7eec00fc1178e41984af7e0905"
server_key_version: "encryptionkey@0"
I20250903 08:07:42.808977 13214 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/minicluster-data/ts-0/wal/instance:
uuid: "ef1366fcc0cf46e9b90f47de268473a9"
format_stamp: "Formatted at 2025-09-03 08:07:42 on dist-test-slave-27r9"
server_key: "66c4c6e84f321deb00b5e56ee231e5ca"
server_key_iv: "02877f7eec00fc1178e41984af7e0905"
server_key_version: "encryptionkey@0"
I20250903 08:07:42.812381 13214 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250903 08:07:42.814522 13229 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:07:42.815529 13214 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250903 08:07:42.815645 13214 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/minicluster-data/ts-0/wal
uuid: "ef1366fcc0cf46e9b90f47de268473a9"
format_stamp: "Formatted at 2025-09-03 08:07:42 on dist-test-slave-27r9"
server_key: "66c4c6e84f321deb00b5e56ee231e5ca"
server_key_iv: "02877f7eec00fc1178e41984af7e0905"
server_key_version: "encryptionkey@0"
I20250903 08:07:42.815737 13214 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:07:42.840375 13214 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:07:42.843057 13214 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:07:42.843232 13214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:07:42.843775 13214 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 08:07:42.844614 13214 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 08:07:42.844666 13214 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 08:07:42.844724 13214 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 08:07:42.844758 13214 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 08:07:42.853492 13214 rpc_server.cc:307] RPC server started. Bound to: 127.12.201.193:36289
I20250903 08:07:42.853529 13342 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.201.193:36289 every 8 connection(s)
I20250903 08:07:42.854414 13214 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/minicluster-data/ts-0/data/info.pb
I20250903 08:07:42.862254 13095 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUML90r/build/debug/bin/kudu as pid 13214
I20250903 08:07:42.862385 13095 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/minicluster-data/ts-0/wal/instance
Sep 03 08:07:42 dist-test-slave-27r9 krb5kdc[13102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886862, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.193@KRBTEST.COM for kudu/127.12.201.254@KRBTEST.COM
I20250903 08:07:42.862627 13095 external_mini_cluster.cc:1468] Setting key 4ceeecc2651837c12a9fcf44c81bcfe0
I20250903 08:07:42.866698 13189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:42.856117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.193:34547 (local address 127.12.201.254:36733)
0903 08:07:42.856256 (+ 139us) server_negotiation.cc:207] Beginning negotiation
0903 08:07:42.856259 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 08:07:42.856877 (+ 618us) server_negotiation.cc:408] Connection header received
0903 08:07:42.857787 (+ 910us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:07:42.857790 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:07:42.857844 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:07:42.857921 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:07:42.859318 (+ 1397us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:42.859820 (+ 502us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:07:42.860752 (+ 932us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:42.860870 (+ 118us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:42.863042 (+ 2172us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:07:42.863065 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:07:42.863068 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:07:42.863090 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:07:42.864853 (+ 1763us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:07:42.865321 (+ 468us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:07:42.865328 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:07:42.865332 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:07:42.865385 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:07:42.865647 (+ 262us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:07:42.865651 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:07:42.865655 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:07:42.865805 (+ 150us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:07:42.865899 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
0903 08:07:42.866381 (+ 482us) server_negotiation.cc:300] Negotiation successful
0903 08:07:42.866510 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20250903 08:07:42.867429 13345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:42.856352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:36733 (local address 127.12.201.193:34547)
0903 08:07:42.856738 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0903 08:07:42.856774 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0903 08:07:42.857582 (+ 808us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:07:42.858099 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:07:42.858108 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:07:42.858514 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:07:42.859093 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:42.859107 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:42.859918 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:07:42.859920 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:07:42.860666 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:42.860673 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:42.860819 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:42.861337 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:07:42.861366 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:07:42.862869 (+ 1503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:07:42.864981 (+ 2112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:07:42.864987 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:07:42.864996 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:07:42.865229 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:07:42.865467 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:07:42.865469 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:07:42.865471 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:07:42.865546 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:07:42.865910 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:07:42.865918 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:07:42.866141 (+ 223us) client_negotiation.cc:769] Sending connection context
0903 08:07:42.866333 (+ 192us) client_negotiation.cc:240] Negotiation successful
0903 08:07:42.866622 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":100,"threads_started":1}
I20250903 08:07:42.868814 13343 heartbeater.cc:344] Connected to a master server at 127.12.201.254:36733
I20250903 08:07:42.869055 13343 heartbeater.cc:461] Registering TS with master...
I20250903 08:07:42.869570 13343 heartbeater.cc:507] Master 127.12.201.254:36733 requested a full tablet report, sending...
I20250903 08:07:42.871271 13150 ts_manager.cc:194] Registered new tserver with Master: ef1366fcc0cf46e9b90f47de268473a9 (127.12.201.193:36289)
WARNING: no policy specified for kudu/127.12.201.194@KRBTEST.COM; defaulting to no policy
I20250903 08:07:42.872625 13150 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.201.193@KRBTEST.COM'} at 127.12.201.193:34547
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.201.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.201.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.201.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.201.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.201.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.201.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.201.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/krb5kdc/kudu.keytab.
I20250903 08:07:42.912076 13095 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUML90r/build/debug/bin/kudu
/tmp/dist-test-taskUML90r/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.201.194:0
--local_ip_for_outbound_sockets=127.12.201.194
--webserver_interface=127.12.201.194
--webserver_port=0
--tserver_master_addrs=127.12.201.254:36733
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.201.212:43327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 08:07:43.017580 13350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:07:43.017802 13350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:07:43.017861 13350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:07:43.021051 13350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 08:07:43.021106 13350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:07:43.021185 13350 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.201.194
I20250903 08:07:43.025223 13350 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.201.212:43327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.201.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.201.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.12.201.254:36733
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13350
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.201.194
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 07:50:08 UTC on 5fd53c4cbb9d
build id 7874
I20250903 08:07:43.026160 13350 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:07:43.027071 13350 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:07:43.033313 13355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:07:43.033341 13358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:07:43.033396 13356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:07:43.033686 13350 server_base.cc:1047] running on GCE node
I20250903 08:07:43.034024 13350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:07:43.034519 13350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:07:43.035665 13350 hybrid_clock.cc:648] HybridClock initialized: now 1756886863035651 us; error 24 us; skew 500 ppm
Sep 03 08:07:43 dist-test-slave-27r9 krb5kdc[13102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886863, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 08:07:43.038232 13350 init.cc:377] Logged in from keytab as kudu/127.12.201.194@KRBTEST.COM (short username kudu)
I20250903 08:07:43.039209 13350 webserver.cc:480] Webserver started at http://127.12.201.194:38031/ using document root <none> and password file <none>
I20250903 08:07:43.039721 13350 fs_manager.cc:362] Metadata directory not provided
I20250903 08:07:43.039764 13350 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:07:43.039913 13350 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:07:43.041528 13350 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/minicluster-data/ts-1/data/instance:
uuid: "ead3708f23164ee8a611e478d93f0bbf"
format_stamp: "Formatted at 2025-09-03 08:07:43 on dist-test-slave-27r9"
server_key: "53b9da3286dc050c4c3e34f819842d87"
server_key_iv: "0406eb4c4cee045480990100c6749578"
server_key_version: "encryptionkey@0"
I20250903 08:07:43.041953 13350 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/minicluster-data/ts-1/wal/instance:
uuid: "ead3708f23164ee8a611e478d93f0bbf"
format_stamp: "Formatted at 2025-09-03 08:07:43 on dist-test-slave-27r9"
server_key: "53b9da3286dc050c4c3e34f819842d87"
server_key_iv: "0406eb4c4cee045480990100c6749578"
server_key_version: "encryptionkey@0"
I20250903 08:07:43.045231 13350 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250903 08:07:43.047305 13365 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:07:43.048270 13350 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250903 08:07:43.048374 13350 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/minicluster-data/ts-1/wal
uuid: "ead3708f23164ee8a611e478d93f0bbf"
format_stamp: "Formatted at 2025-09-03 08:07:43 on dist-test-slave-27r9"
server_key: "53b9da3286dc050c4c3e34f819842d87"
server_key_iv: "0406eb4c4cee045480990100c6749578"
server_key_version: "encryptionkey@0"
I20250903 08:07:43.048465 13350 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:07:43.059908 13350 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:07:43.062659 13350 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:07:43.062824 13350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:07:43.063360 13350 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 08:07:43.064170 13350 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 08:07:43.064220 13350 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 08:07:43.064283 13350 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 08:07:43.064312 13350 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 08:07:43.073985 13350 rpc_server.cc:307] RPC server started. Bound to: 127.12.201.194:37885
I20250903 08:07:43.074029 13478 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.201.194:37885 every 8 connection(s)
I20250903 08:07:43.074930 13350 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/minicluster-data/ts-1/data/info.pb
I20250903 08:07:43.077203 13095 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUML90r/build/debug/bin/kudu as pid 13350
I20250903 08:07:43.077309 13095 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/minicluster-data/ts-1/wal/instance
I20250903 08:07:43.077543 13095 external_mini_cluster.cc:1468] Setting key 7993f018acf62f2666141ed233ae07ad
Sep 03 08:07:43 dist-test-slave-27r9 krb5kdc[13102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886863, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.194@KRBTEST.COM for kudu/127.12.201.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.201.195@KRBTEST.COM; defaulting to no policy
I20250903 08:07:43.087561 13189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:43.076723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.194:45255 (local address 127.12.201.254:36733)
0903 08:07:43.076902 (+ 179us) server_negotiation.cc:207] Beginning negotiation
0903 08:07:43.076906 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:07:43.077586 (+ 680us) server_negotiation.cc:408] Connection header received
0903 08:07:43.078532 (+ 946us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:07:43.078534 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:07:43.078579 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:07:43.078653 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:07:43.079993 (+ 1340us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:43.080784 (+ 791us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:07:43.081891 (+ 1107us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:43.082135 (+ 244us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:43.084324 (+ 2189us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:07:43.084340 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:07:43.084343 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:07:43.084377 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:07:43.085754 (+ 1377us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:07:43.086282 (+ 528us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:07:43.086285 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:07:43.086286 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:07:43.086325 (+ 39us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:07:43.086555 (+ 230us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:07:43.086557 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:07:43.086558 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:07:43.086687 (+ 129us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:07:43.086758 (+ 71us) server_negotiation.cc:1036] Waiting for connection context
0903 08:07:43.087321 (+ 563us) server_negotiation.cc:300] Negotiation successful
0903 08:07:43.087439 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20250903 08:07:43.088192 13481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:43.076952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:36733 (local address 127.12.201.194:45255)
0903 08:07:43.077441 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0903 08:07:43.077479 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0903 08:07:43.078331 (+ 852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:07:43.078837 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:07:43.078850 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:07:43.079303 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:07:43.079844 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:43.079856 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:43.080936 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:07:43.080967 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:07:43.081774 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:43.081783 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:43.081949 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:43.082567 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:07:43.082589 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:07:43.084164 (+ 1575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:07:43.085882 (+ 1718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:07:43.085891 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:07:43.085905 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:07:43.086204 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:07:43.086405 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:07:43.086407 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:07:43.086409 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:07:43.086482 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:07:43.086799 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:07:43.086804 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:07:43.087067 (+ 263us) client_negotiation.cc:769] Sending connection context
0903 08:07:43.087264 (+ 197us) client_negotiation.cc:240] Negotiation successful
0903 08:07:43.087482 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":128,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.201.195@KRBTEST.COM" created.
I20250903 08:07:43.089293 13479 heartbeater.cc:344] Connected to a master server at 127.12.201.254:36733
I20250903 08:07:43.089542 13479 heartbeater.cc:461] Registering TS with master...
I20250903 08:07:43.090034 13479 heartbeater.cc:507] Master 127.12.201.254:36733 requested a full tablet report, sending...
I20250903 08:07:43.091094 13150 ts_manager.cc:194] Registered new tserver with Master: ead3708f23164ee8a611e478d93f0bbf (127.12.201.194:37885)
I20250903 08:07:43.091645 13150 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.201.194@KRBTEST.COM'} at 127.12.201.194:45255
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.201.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.201.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.201.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.201.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.201.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.201.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/krb5kdc/kudu.keytab.
I20250903 08:07:43.127494 13095 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUML90r/build/debug/bin/kudu
/tmp/dist-test-taskUML90r/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.201.195:0
--local_ip_for_outbound_sockets=127.12.201.195
--webserver_interface=127.12.201.195
--webserver_port=0
--tserver_master_addrs=127.12.201.254:36733
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.201.212:43327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 08:07:43.235214 13486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:07:43.235472 13486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:07:43.235562 13486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:07:43.238850 13486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 08:07:43.238929 13486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:07:43.239082 13486 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.201.195
I20250903 08:07:43.243479 13486 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.201.212:43327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.201.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.201.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.12.201.254:36733
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13486
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.201.195
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 07:50:08 UTC on 5fd53c4cbb9d
build id 7874
I20250903 08:07:43.244585 13486 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:07:43.245434 13486 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:07:43.251816 13492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:07:43.251834 13494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:07:43.251927 13491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:07:43.252255 13486 server_base.cc:1047] running on GCE node
I20250903 08:07:43.252682 13486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:07:43.253377 13486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:07:43.254528 13486 hybrid_clock.cc:648] HybridClock initialized: now 1756886863254505 us; error 35 us; skew 500 ppm
Sep 03 08:07:43 dist-test-slave-27r9 krb5kdc[13102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886863, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 08:07:43.257427 13486 init.cc:377] Logged in from keytab as kudu/127.12.201.195@KRBTEST.COM (short username kudu)
I20250903 08:07:43.258423 13486 webserver.cc:480] Webserver started at http://127.12.201.195:42869/ using document root <none> and password file <none>
I20250903 08:07:43.258965 13486 fs_manager.cc:362] Metadata directory not provided
I20250903 08:07:43.259058 13486 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:07:43.259236 13486 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:07:43.260926 13486 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/minicluster-data/ts-2/data/instance:
uuid: "dd0471e2ffbc4f9ab3c62fa4976d566a"
format_stamp: "Formatted at 2025-09-03 08:07:43 on dist-test-slave-27r9"
server_key: "138bc370b3dbe0e28b65b6c428fe3c84"
server_key_iv: "614e4a7fdc5cfe8d7ad012c65f6e8bf4"
server_key_version: "encryptionkey@0"
I20250903 08:07:43.261408 13486 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/minicluster-data/ts-2/wal/instance:
uuid: "dd0471e2ffbc4f9ab3c62fa4976d566a"
format_stamp: "Formatted at 2025-09-03 08:07:43 on dist-test-slave-27r9"
server_key: "138bc370b3dbe0e28b65b6c428fe3c84"
server_key_iv: "614e4a7fdc5cfe8d7ad012c65f6e8bf4"
server_key_version: "encryptionkey@0"
I20250903 08:07:43.264711 13486 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20250903 08:07:43.266948 13501 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:07:43.267993 13486 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250903 08:07:43.268095 13486 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/minicluster-data/ts-2/wal
uuid: "dd0471e2ffbc4f9ab3c62fa4976d566a"
format_stamp: "Formatted at 2025-09-03 08:07:43 on dist-test-slave-27r9"
server_key: "138bc370b3dbe0e28b65b6c428fe3c84"
server_key_iv: "614e4a7fdc5cfe8d7ad012c65f6e8bf4"
server_key_version: "encryptionkey@0"
I20250903 08:07:43.268182 13486 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:07:43.292212 13486 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:07:43.295245 13486 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:07:43.295420 13486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:07:43.295967 13486 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 08:07:43.296805 13486 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 08:07:43.296854 13486 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 08:07:43.296938 13486 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 08:07:43.296972 13486 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 08:07:43.306703 13486 rpc_server.cc:307] RPC server started. Bound to: 127.12.201.195:36469
I20250903 08:07:43.306802 13614 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.201.195:36469 every 8 connection(s)
I20250903 08:07:43.307938 13486 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/minicluster-data/ts-2/data/info.pb
I20250903 08:07:43.313234 13095 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUML90r/build/debug/bin/kudu as pid 13486
I20250903 08:07:43.313373 13095 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756886860253814-13095-0/minicluster-data/ts-2/wal/instance
I20250903 08:07:43.313776 13095 external_mini_cluster.cc:1468] Setting key 39a1e95a99f1cac8a14f9cee02d416ae
Sep 03 08:07:43 dist-test-slave-27r9 krb5kdc[13102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886863, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.195@KRBTEST.COM for kudu/127.12.201.254@KRBTEST.COM
I20250903 08:07:43.321763 13189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:43.309749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.195:51455 (local address 127.12.201.254:36733)
0903 08:07:43.309873 (+ 124us) server_negotiation.cc:207] Beginning negotiation
0903 08:07:43.309876 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 08:07:43.310608 (+ 732us) server_negotiation.cc:408] Connection header received
0903 08:07:43.311626 (+ 1018us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:07:43.311629 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:07:43.311680 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:07:43.311768 (+ 88us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:07:43.314105 (+ 2337us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:43.314775 (+ 670us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:07:43.316019 (+ 1244us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:43.316234 (+ 215us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:43.318489 (+ 2255us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:07:43.318505 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:07:43.318508 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:07:43.318534 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:07:43.319985 (+ 1451us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:07:43.320510 (+ 525us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:07:43.320513 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:07:43.320514 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:07:43.320554 (+ 40us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:07:43.320763 (+ 209us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:07:43.320765 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:07:43.320766 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:07:43.320901 (+ 135us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:07:43.320973 (+ 72us) server_negotiation.cc:1036] Waiting for connection context
0903 08:07:43.321521 (+ 548us) server_negotiation.cc:300] Negotiation successful
0903 08:07:43.321638 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20250903 08:07:43.322419 13617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:43.309975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:36733 (local address 127.12.201.195:51455)
0903 08:07:43.310451 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0903 08:07:43.310490 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0903 08:07:43.311355 (+ 865us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:07:43.311973 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:07:43.311984 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:07:43.313028 (+ 1044us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:07:43.313899 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:43.313920 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:43.314918 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:07:43.314922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:07:43.315888 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:43.315896 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:43.316065 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:43.316758 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:07:43.316783 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:07:43.318319 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:07:43.320148 (+ 1829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:07:43.320153 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:07:43.320163 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:07:43.320422 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:07:43.320620 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:07:43.320622 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:07:43.320624 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:07:43.320698 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:07:43.321028 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:07:43.321034 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:07:43.321267 (+ 233us) client_negotiation.cc:769] Sending connection context
0903 08:07:43.321465 (+ 198us) client_negotiation.cc:240] Negotiation successful
0903 08:07:43.321696 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":102,"threads_started":1}
I20250903 08:07:43.323566 13615 heartbeater.cc:344] Connected to a master server at 127.12.201.254:36733
I20250903 08:07:43.323825 13615 heartbeater.cc:461] Registering TS with master...
I20250903 08:07:43.324362 13615 heartbeater.cc:507] Master 127.12.201.254:36733 requested a full tablet report, sending...
I20250903 08:07:43.325330 13150 ts_manager.cc:194] Registered new tserver with Master: dd0471e2ffbc4f9ab3c62fa4976d566a (127.12.201.195:36469)
I20250903 08:07:43.325907 13150 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.201.195@KRBTEST.COM'} at 127.12.201.195:51455
I20250903 08:07:43.328265 13095 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Sep 03 08:07:43 dist-test-slave-27r9 krb5kdc[13102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886863, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 03 08:07:43 dist-test-slave-27r9 krb5kdc[13102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886863, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.201.193@KRBTEST.COM
I20250903 08:07:43.351809 13622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:43.343068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35594 (local address 127.12.201.193:36289)
0903 08:07:43.343391 (+ 323us) server_negotiation.cc:207] Beginning negotiation
0903 08:07:43.343397 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0903 08:07:43.343421 (+ 24us) server_negotiation.cc:408] Connection header received
0903 08:07:43.343475 (+ 54us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:07:43.343480 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:07:43.343659 (+ 179us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:07:43.343787 (+ 128us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:07:43.344717 (+ 930us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:43.345410 (+ 693us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:07:43.346026 (+ 616us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:43.346185 (+ 159us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:43.348483 (+ 2298us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:07:43.348511 (+ 28us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:07:43.348522 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:07:43.348548 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:07:43.350236 (+ 1688us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:07:43.350658 (+ 422us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:07:43.350663 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:07:43.350664 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:07:43.350714 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:07:43.350944 (+ 230us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:07:43.350946 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:07:43.350948 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:07:43.351169 (+ 221us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:07:43.351332 (+ 163us) server_negotiation.cc:1036] Waiting for connection context
0903 08:07:43.351519 (+ 187us) server_negotiation.cc:300] Negotiation successful
0903 08:07:43.351652 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":127,"threads_started":1}
W20250903 08:07:43.352744 13257 server_base.cc:1129] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35594
Sep 03 08:07:43 dist-test-slave-27r9 krb5kdc[13102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 03 08:07:43 dist-test-slave-27r9 krb5kdc[13102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.201.193@KRBTEST.COM
I20250903 08:07:43.376214 13622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:43.368608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35604 (local address 127.12.201.193:36289)
0903 08:07:43.368802 (+ 194us) server_negotiation.cc:207] Beginning negotiation
0903 08:07:43.368807 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 08:07:43.368891 (+ 84us) server_negotiation.cc:408] Connection header received
0903 08:07:43.369054 (+ 163us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:07:43.369057 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:07:43.369106 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:07:43.369181 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:07:43.369986 (+ 805us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:43.370508 (+ 522us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:07:43.371156 (+ 648us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:43.371377 (+ 221us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:43.373548 (+ 2171us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:07:43.373568 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:07:43.373571 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:07:43.373595 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:07:43.374950 (+ 1355us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:07:43.375395 (+ 445us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:07:43.375398 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:07:43.375399 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:07:43.375444 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:07:43.375612 (+ 168us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:07:43.375615 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:07:43.375616 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:07:43.375743 (+ 127us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:07:43.375838 (+ 95us) server_negotiation.cc:1036] Waiting for connection context
0903 08:07:43.375993 (+ 155us) server_negotiation.cc:300] Negotiation successful
0903 08:07:43.376088 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20250903 08:07:43.377182 13095 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUML90r/build/debug/bin/kudu with pid 13214
I20250903 08:07:43.382818 13095 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUML90r/build/debug/bin/kudu with pid 13350
I20250903 08:07:43.390206 13095 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUML90r/build/debug/bin/kudu with pid 13486
I20250903 08:07:43.396047 13095 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUML90r/build/debug/bin/kudu with pid 13118
2025-09-03T08:07:43Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3153 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 03 08:07:43 dist-test-slave-27r9 krb5kdc[13629](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 03 08:07:43 dist-test-slave-27r9 krb5kdc[13629](info): set up 2 sockets
Sep 03 08:07:43 dist-test-slave-27r9 krb5kdc[13629](info): commencing operation
krb5kdc: starting...
W20250903 08:07:45.387198 13095 mini_kdc.cc:121] Time spent starting KDC: real 1.969s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/krb5kdc/test-user.keytab.
Sep 03 08:07:45 dist-test-slave-27r9 krb5kdc[13629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886865, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-09-03T08:07:45Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-03T08:07:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.201.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.201.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.201.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.201.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.201.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.201.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.201.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.201.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/krb5kdc/kudu.keytab.
I20250903 08:07:45.521344 13095 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUML90r/build/debug/bin/kudu
/tmp/dist-test-taskUML90r/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.201.254:40795
--webserver_interface=127.12.201.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.201.212:39179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.201.254:40795
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 08:07:45.625128 13645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:07:45.625388 13645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:07:45.625485 13645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:07:45.628755 13645 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250903 08:07:45.628827 13645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 08:07:45.628865 13645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:07:45.628898 13645 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250903 08:07:45.628939 13645 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250903 08:07:45.633239 13645 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.201.212:39179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.201.254:40795
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.201.254:40795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.201.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13645
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 07:50:08 UTC on 5fd53c4cbb9d
build id 7874
I20250903 08:07:45.634362 13645 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:07:45.635272 13645 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:07:45.640723 13650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:07:45.640784 13651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:07:45.640738 13653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:07:45.641039 13645 server_base.cc:1047] running on GCE node
I20250903 08:07:45.641518 13645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:07:45.642419 13645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:07:45.643594 13645 hybrid_clock.cc:648] HybridClock initialized: now 1756886865643563 us; error 27 us; skew 500 ppm
Sep 03 08:07:45 dist-test-slave-27r9 krb5kdc[13629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886865, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 08:07:45.646239 13645 init.cc:377] Logged in from keytab as kudu/127.12.201.254@KRBTEST.COM (short username kudu)
I20250903 08:07:45.647356 13645 webserver.cc:480] Webserver started at http://127.12.201.254:44919/ using document root <none> and password file <none>
I20250903 08:07:45.647907 13645 fs_manager.cc:362] Metadata directory not provided
I20250903 08:07:45.647962 13645 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:07:45.648152 13645 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:07:45.649914 13645 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/minicluster-data/master-0/data/instance:
uuid: "fd925dd74272462fb6d72de2457b451c"
format_stamp: "Formatted at 2025-09-03 08:07:45 on dist-test-slave-27r9"
server_key: "5c17331ee326e6ace22dd5bc389c16df"
server_key_iv: "ec446a0476ec7f1e88df1bc088176354"
server_key_version: "encryptionkey@0"
I20250903 08:07:45.650401 13645 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/minicluster-data/master-0/wal/instance:
uuid: "fd925dd74272462fb6d72de2457b451c"
format_stamp: "Formatted at 2025-09-03 08:07:45 on dist-test-slave-27r9"
server_key: "5c17331ee326e6ace22dd5bc389c16df"
server_key_iv: "ec446a0476ec7f1e88df1bc088176354"
server_key_version: "encryptionkey@0"
I20250903 08:07:45.653863 13645 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20250903 08:07:45.656162 13660 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:07:45.657162 13645 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250903 08:07:45.657276 13645 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/minicluster-data/master-0/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/minicluster-data/master-0/wal
uuid: "fd925dd74272462fb6d72de2457b451c"
format_stamp: "Formatted at 2025-09-03 08:07:45 on dist-test-slave-27r9"
server_key: "5c17331ee326e6ace22dd5bc389c16df"
server_key_iv: "ec446a0476ec7f1e88df1bc088176354"
server_key_version: "encryptionkey@0"
I20250903 08:07:45.657379 13645 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:07:45.671223 13645 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:07:45.679631 13645 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:07:45.679844 13645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:07:45.687294 13712 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.201.254:40795 every 8 connection(s)
I20250903 08:07:45.687307 13645 rpc_server.cc:307] RPC server started. Bound to: 127.12.201.254:40795
I20250903 08:07:45.688359 13645 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/minicluster-data/master-0/data/info.pb
I20250903 08:07:45.690835 13713 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:07:45.696089 13713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd925dd74272462fb6d72de2457b451c: Bootstrap starting.
I20250903 08:07:45.696879 13095 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUML90r/build/debug/bin/kudu as pid 13645
I20250903 08:07:45.696978 13095 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/minicluster-data/master-0/wal/instance
I20250903 08:07:45.697176 13095 external_mini_cluster.cc:1468] Setting key 763d1934c90ccc86c807ff9612b63cf5
I20250903 08:07:45.698494 13713 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd925dd74272462fb6d72de2457b451c: Neither blocks nor log segments found. Creating new log.
I20250903 08:07:45.699193 13713 log.cc:826] T 00000000000000000000000000000000 P fd925dd74272462fb6d72de2457b451c: Log is configured to *not* fsync() on all Append() calls
I20250903 08:07:45.700824 13713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd925dd74272462fb6d72de2457b451c: No bootstrap required, opened a new log
I20250903 08:07:45.703316 13713 raft_consensus.cc:357] T 00000000000000000000000000000000 P fd925dd74272462fb6d72de2457b451c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd925dd74272462fb6d72de2457b451c" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 40795 } }
I20250903 08:07:45.703544 13713 raft_consensus.cc:383] T 00000000000000000000000000000000 P fd925dd74272462fb6d72de2457b451c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:07:45.703616 13713 raft_consensus.cc:738] T 00000000000000000000000000000000 P fd925dd74272462fb6d72de2457b451c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd925dd74272462fb6d72de2457b451c, State: Initialized, Role: FOLLOWER
Sep 03 08:07:45 dist-test-slave-27r9 krb5kdc[13629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886865, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.201.254@KRBTEST.COM
I20250903 08:07:45.704077 13713 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd925dd74272462fb6d72de2457b451c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd925dd74272462fb6d72de2457b451c" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 40795 } }
I20250903 08:07:45.704210 13713 raft_consensus.cc:397] T 00000000000000000000000000000000 P fd925dd74272462fb6d72de2457b451c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250903 08:07:45.704282 13713 raft_consensus.cc:491] T 00000000000000000000000000000000 P fd925dd74272462fb6d72de2457b451c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250903 08:07:45.704384 13713 raft_consensus.cc:3058] T 00000000000000000000000000000000 P fd925dd74272462fb6d72de2457b451c [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:07:45.705314 13713 raft_consensus.cc:513] T 00000000000000000000000000000000 P fd925dd74272462fb6d72de2457b451c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd925dd74272462fb6d72de2457b451c" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 40795 } }
I20250903 08:07:45.705670 13713 leader_election.cc:304] T 00000000000000000000000000000000 P fd925dd74272462fb6d72de2457b451c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd925dd74272462fb6d72de2457b451c; no voters:
I20250903 08:07:45.705935 13713 leader_election.cc:290] T 00000000000000000000000000000000 P fd925dd74272462fb6d72de2457b451c [CANDIDATE]: Term 1 election: Requested vote from peers
I20250903 08:07:45.706305 13718 raft_consensus.cc:2802] T 00000000000000000000000000000000 P fd925dd74272462fb6d72de2457b451c [term 1 FOLLOWER]: Leader election won for term 1
I20250903 08:07:45.706513 13718 raft_consensus.cc:695] T 00000000000000000000000000000000 P fd925dd74272462fb6d72de2457b451c [term 1 LEADER]: Becoming Leader. State: Replica: fd925dd74272462fb6d72de2457b451c, State: Running, Role: LEADER
I20250903 08:07:45.706817 13718 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd925dd74272462fb6d72de2457b451c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd925dd74272462fb6d72de2457b451c" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 40795 } }
I20250903 08:07:45.707085 13713 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd925dd74272462fb6d72de2457b451c [sys.catalog]: configured and running, proceeding with master startup.
I20250903 08:07:45.708297 13720 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd925dd74272462fb6d72de2457b451c [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd925dd74272462fb6d72de2457b451c. Latest consensus state: current_term: 1 leader_uuid: "fd925dd74272462fb6d72de2457b451c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd925dd74272462fb6d72de2457b451c" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 40795 } } }
I20250903 08:07:45.708398 13720 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd925dd74272462fb6d72de2457b451c [sys.catalog]: This master's current role is: LEADER
I20250903 08:07:45.708699 13719 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd925dd74272462fb6d72de2457b451c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd925dd74272462fb6d72de2457b451c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd925dd74272462fb6d72de2457b451c" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 40795 } } }
I20250903 08:07:45.708778 13719 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd925dd74272462fb6d72de2457b451c [sys.catalog]: This master's current role is: LEADER
I20250903 08:07:45.709771 13716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:45.698129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39358 (local address 127.12.201.254:40795)
0903 08:07:45.698492 (+ 363us) server_negotiation.cc:207] Beginning negotiation
0903 08:07:45.698500 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0903 08:07:45.698534 (+ 34us) server_negotiation.cc:408] Connection header received
0903 08:07:45.699106 (+ 572us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:07:45.699123 (+ 17us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:07:45.699432 (+ 309us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:07:45.699720 (+ 288us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:07:45.700576 (+ 856us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:45.701347 (+ 771us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:07:45.701958 (+ 611us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:45.702246 (+ 288us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:45.704601 (+ 2355us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:07:45.704620 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:07:45.704630 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:07:45.704662 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:07:45.706737 (+ 2075us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:07:45.707216 (+ 479us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:07:45.707222 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:07:45.707228 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:07:45.707326 (+ 98us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:07:45.707637 (+ 311us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:07:45.707641 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:07:45.707643 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:07:45.708084 (+ 441us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:07:45.708294 (+ 210us) server_negotiation.cc:1036] Waiting for connection context
0903 08:07:45.708759 (+ 465us) server_negotiation.cc:300] Negotiation successful
0903 08:07:45.709023 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":109,"threads_started":1}
W20250903 08:07:45.711799 13733 catalog_manager.cc:1560] T 00000000000000000000000000000000 P fd925dd74272462fb6d72de2457b451c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250903 08:07:45.711872 13733 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250903 08:07:45.711941 13727 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250903 08:07:45.712800 13727 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250903 08:07:45.717932 13727 catalog_manager.cc:1349] Generated new cluster ID: a7498f774c14447f8742875a810f01ca
I20250903 08:07:45.717998 13727 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250903 08:07:45.733316 13727 catalog_manager.cc:1372] Generated new certificate authority record
I20250903 08:07:45.734364 13727 catalog_manager.cc:1506] Loading token signing keys...
I20250903 08:07:45.739384 13727 catalog_manager.cc:5955] T 00000000000000000000000000000000 P fd925dd74272462fb6d72de2457b451c: Generated new TSK 0
I20250903 08:07:45.740041 13727 catalog_manager.cc:1516] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.201.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.201.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.201.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.201.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.201.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.201.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.201.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.201.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/krb5kdc/kudu.keytab.
I20250903 08:07:45.786801 13095 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUML90r/build/debug/bin/kudu
/tmp/dist-test-taskUML90r/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.201.193:0
--local_ip_for_outbound_sockets=127.12.201.193
--webserver_interface=127.12.201.193
--webserver_port=0
--tserver_master_addrs=127.12.201.254:40795
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.201.212:39179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 08:07:45.893146 13741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:07:45.893368 13741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:07:45.893428 13741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:07:45.896580 13741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 08:07:45.896637 13741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:07:45.896715 13741 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.201.193
I20250903 08:07:45.900768 13741 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.201.212:39179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.201.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.201.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.12.201.254:40795
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13741
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.201.193
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 07:50:08 UTC on 5fd53c4cbb9d
build id 7874
I20250903 08:07:45.901757 13741 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:07:45.902551 13741 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:07:45.908615 13746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:07:45.908617 13749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:07:45.908730 13747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:07:45.908936 13741 server_base.cc:1047] running on GCE node
I20250903 08:07:45.909317 13741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:07:45.909819 13741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:07:45.910964 13741 hybrid_clock.cc:648] HybridClock initialized: now 1756886865910950 us; error 34 us; skew 500 ppm
Sep 03 08:07:45 dist-test-slave-27r9 krb5kdc[13629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886865, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 08:07:45.913520 13741 init.cc:377] Logged in from keytab as kudu/127.12.201.193@KRBTEST.COM (short username kudu)
I20250903 08:07:45.914433 13741 webserver.cc:480] Webserver started at http://127.12.201.193:45645/ using document root <none> and password file <none>
I20250903 08:07:45.914947 13741 fs_manager.cc:362] Metadata directory not provided
I20250903 08:07:45.915035 13741 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:07:45.915230 13741 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:07:45.916833 13741 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/minicluster-data/ts-0/data/instance:
uuid: "6209003669834836875868588d559ce9"
format_stamp: "Formatted at 2025-09-03 08:07:45 on dist-test-slave-27r9"
server_key: "c017489ca850eb3bc0851531d37c134d"
server_key_iv: "97096deb8ed52506dd1ed64ef7cfb0a7"
server_key_version: "encryptionkey@0"
I20250903 08:07:45.917272 13741 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/minicluster-data/ts-0/wal/instance:
uuid: "6209003669834836875868588d559ce9"
format_stamp: "Formatted at 2025-09-03 08:07:45 on dist-test-slave-27r9"
server_key: "c017489ca850eb3bc0851531d37c134d"
server_key_iv: "97096deb8ed52506dd1ed64ef7cfb0a7"
server_key_version: "encryptionkey@0"
I20250903 08:07:45.920331 13741 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20250903 08:07:45.922273 13756 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:07:45.923255 13741 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20250903 08:07:45.923354 13741 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/minicluster-data/ts-0/wal
uuid: "6209003669834836875868588d559ce9"
format_stamp: "Formatted at 2025-09-03 08:07:45 on dist-test-slave-27r9"
server_key: "c017489ca850eb3bc0851531d37c134d"
server_key_iv: "97096deb8ed52506dd1ed64ef7cfb0a7"
server_key_version: "encryptionkey@0"
I20250903 08:07:45.923436 13741 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:07:45.946369 13741 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:07:45.949255 13741 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:07:45.949452 13741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:07:45.950017 13741 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 08:07:45.950868 13741 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 08:07:45.950918 13741 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 08:07:45.950999 13741 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 08:07:45.951035 13741 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 08:07:45.959867 13741 rpc_server.cc:307] RPC server started. Bound to: 127.12.201.193:41851
I20250903 08:07:45.959894 13869 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.201.193:41851 every 8 connection(s)
I20250903 08:07:45.960857 13741 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/minicluster-data/ts-0/data/info.pb
I20250903 08:07:45.962190 13095 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUML90r/build/debug/bin/kudu as pid 13741
I20250903 08:07:45.962329 13095 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/minicluster-data/ts-0/wal/instance
I20250903 08:07:45.962571 13095 external_mini_cluster.cc:1468] Setting key ea3d62b6827ac111eaaf3f1bf9563967
Sep 03 08:07:45 dist-test-slave-27r9 krb5kdc[13629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886865, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.193@KRBTEST.COM for kudu/127.12.201.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.201.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.201.194@KRBTEST.COM" created.
I20250903 08:07:45.975457 13716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:45.962838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.193:34313 (local address 127.12.201.254:40795)
0903 08:07:45.963016 (+ 178us) server_negotiation.cc:207] Beginning negotiation
0903 08:07:45.963020 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:07:45.963805 (+ 785us) server_negotiation.cc:408] Connection header received
0903 08:07:45.964728 (+ 923us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:07:45.964732 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:07:45.964785 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:07:45.964865 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:07:45.966582 (+ 1717us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:45.967159 (+ 577us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:07:45.968887 (+ 1728us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:45.969029 (+ 142us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:45.971799 (+ 2770us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:07:45.971821 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:07:45.971823 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:07:45.971854 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:07:45.973328 (+ 1474us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:07:45.973826 (+ 498us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:07:45.973829 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:07:45.973831 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:07:45.973874 (+ 43us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:07:45.974084 (+ 210us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:07:45.974086 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:07:45.974088 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:07:45.974316 (+ 228us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:07:45.974416 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
0903 08:07:45.975020 (+ 604us) server_negotiation.cc:300] Negotiation successful
0903 08:07:45.975171 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20250903 08:07:45.976049 13872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:45.963153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:40795 (local address 127.12.201.193:34313)
0903 08:07:45.963673 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0903 08:07:45.963722 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0903 08:07:45.964530 (+ 808us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:07:45.965029 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:07:45.965037 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:07:45.965585 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:07:45.966399 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:45.966422 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:45.967325 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:07:45.967332 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:07:45.968742 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:45.968757 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:45.969011 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:45.969697 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:07:45.969739 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:07:45.971617 (+ 1878us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:07:45.973444 (+ 1827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:07:45.973449 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:07:45.973459 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:07:45.973707 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:07:45.973943 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:07:45.973945 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:07:45.973947 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:07:45.974022 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:07:45.974425 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:07:45.974435 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:07:45.974742 (+ 307us) client_negotiation.cc:769] Sending connection context
0903 08:07:45.974937 (+ 195us) client_negotiation.cc:240] Negotiation successful
0903 08:07:45.975247 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":110,"threads_started":1}
I20250903 08:07:45.977231 13870 heartbeater.cc:344] Connected to a master server at 127.12.201.254:40795
I20250903 08:07:45.977522 13870 heartbeater.cc:461] Registering TS with master...
I20250903 08:07:45.978178 13870 heartbeater.cc:507] Master 127.12.201.254:40795 requested a full tablet report, sending...
I20250903 08:07:45.979823 13677 ts_manager.cc:194] Registered new tserver with Master: 6209003669834836875868588d559ce9 (127.12.201.193:41851)
I20250903 08:07:45.980979 13677 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.201.193@KRBTEST.COM'} at 127.12.201.193:34313
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.201.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.201.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.201.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.201.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.201.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.201.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/krb5kdc/kudu.keytab.
I20250903 08:07:46.010105 13095 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUML90r/build/debug/bin/kudu
/tmp/dist-test-taskUML90r/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.201.194:0
--local_ip_for_outbound_sockets=127.12.201.194
--webserver_interface=127.12.201.194
--webserver_port=0
--tserver_master_addrs=127.12.201.254:40795
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.201.212:39179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 08:07:46.114107 13877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:07:46.114347 13877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:07:46.114430 13877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:07:46.117723 13877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 08:07:46.117792 13877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:07:46.117903 13877 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.201.194
I20250903 08:07:46.121955 13877 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.201.212:39179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.201.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.201.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.12.201.254:40795
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13877
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.201.194
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 07:50:08 UTC on 5fd53c4cbb9d
build id 7874
I20250903 08:07:46.123028 13877 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:07:46.123865 13877 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:07:46.130182 13883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:07:46.130167 13882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:07:46.130169 13885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:07:46.130676 13877 server_base.cc:1047] running on GCE node
I20250903 08:07:46.131088 13877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:07:46.131660 13877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:07:46.132830 13877 hybrid_clock.cc:648] HybridClock initialized: now 1756886866132802 us; error 40 us; skew 500 ppm
Sep 03 08:07:46 dist-test-slave-27r9 krb5kdc[13629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886866, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 08:07:46.135668 13877 init.cc:377] Logged in from keytab as kudu/127.12.201.194@KRBTEST.COM (short username kudu)
I20250903 08:07:46.136690 13877 webserver.cc:480] Webserver started at http://127.12.201.194:33491/ using document root <none> and password file <none>
I20250903 08:07:46.137220 13877 fs_manager.cc:362] Metadata directory not provided
I20250903 08:07:46.137290 13877 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:07:46.137470 13877 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:07:46.139190 13877 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/minicluster-data/ts-1/data/instance:
uuid: "23477b2f0bc846249af230723bdea921"
format_stamp: "Formatted at 2025-09-03 08:07:46 on dist-test-slave-27r9"
server_key: "c31bf99aeadfdfd9484c99a04952c260"
server_key_iv: "49eea1a933f13beaaf3b9add83885044"
server_key_version: "encryptionkey@0"
I20250903 08:07:46.139662 13877 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/minicluster-data/ts-1/wal/instance:
uuid: "23477b2f0bc846249af230723bdea921"
format_stamp: "Formatted at 2025-09-03 08:07:46 on dist-test-slave-27r9"
server_key: "c31bf99aeadfdfd9484c99a04952c260"
server_key_iv: "49eea1a933f13beaaf3b9add83885044"
server_key_version: "encryptionkey@0"
I20250903 08:07:46.142887 13877 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20250903 08:07:46.145030 13892 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:07:46.145920 13877 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250903 08:07:46.146023 13877 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/minicluster-data/ts-1/wal
uuid: "23477b2f0bc846249af230723bdea921"
format_stamp: "Formatted at 2025-09-03 08:07:46 on dist-test-slave-27r9"
server_key: "c31bf99aeadfdfd9484c99a04952c260"
server_key_iv: "49eea1a933f13beaaf3b9add83885044"
server_key_version: "encryptionkey@0"
I20250903 08:07:46.146100 13877 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:07:46.165782 13877 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:07:46.168540 13877 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:07:46.168713 13877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:07:46.169256 13877 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 08:07:46.170061 13877 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 08:07:46.170117 13877 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 08:07:46.170192 13877 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 08:07:46.170225 13877 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 08:07:46.179338 13877 rpc_server.cc:307] RPC server started. Bound to: 127.12.201.194:33221
I20250903 08:07:46.179386 14005 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.201.194:33221 every 8 connection(s)
I20250903 08:07:46.180356 13877 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/minicluster-data/ts-1/data/info.pb
I20250903 08:07:46.185366 13095 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUML90r/build/debug/bin/kudu as pid 13877
I20250903 08:07:46.185482 13095 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/minicluster-data/ts-1/wal/instance
I20250903 08:07:46.185707 13095 external_mini_cluster.cc:1468] Setting key e931d3b0c0f5f5f36266b38a6378e84a
Sep 03 08:07:46 dist-test-slave-27r9 krb5kdc[13629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886866, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.194@KRBTEST.COM for kudu/127.12.201.254@KRBTEST.COM
I20250903 08:07:46.193400 13716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:46.182014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.194:48255 (local address 127.12.201.254:40795)
0903 08:07:46.182147 (+ 133us) server_negotiation.cc:207] Beginning negotiation
0903 08:07:46.182150 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 08:07:46.182872 (+ 722us) server_negotiation.cc:408] Connection header received
0903 08:07:46.183739 (+ 867us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:07:46.183742 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:07:46.183803 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:07:46.183882 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:07:46.185139 (+ 1257us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:46.185697 (+ 558us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:07:46.186882 (+ 1185us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:46.187079 (+ 197us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:46.189667 (+ 2588us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:07:46.189684 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:07:46.189686 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:07:46.189711 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:07:46.191454 (+ 1743us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:07:46.191937 (+ 483us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:07:46.191944 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:07:46.191947 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:07:46.192010 (+ 63us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:07:46.192256 (+ 246us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:07:46.192261 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:07:46.192265 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:07:46.192487 (+ 222us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:07:46.192596 (+ 109us) server_negotiation.cc:1036] Waiting for connection context
0903 08:07:46.193039 (+ 443us) server_negotiation.cc:300] Negotiation successful
0903 08:07:46.193179 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20250903 08:07:46.193881 14008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:46.182285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:40795 (local address 127.12.201.194:48255)
0903 08:07:46.182723 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0903 08:07:46.182761 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0903 08:07:46.183538 (+ 777us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:07:46.184046 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:07:46.184054 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:07:46.184446 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:07:46.184976 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:46.184995 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:46.185814 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:07:46.185819 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:07:46.186773 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:46.186782 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:46.187015 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:46.187695 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:07:46.187720 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:07:46.189507 (+ 1787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:07:46.191579 (+ 2072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:07:46.191584 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:07:46.191594 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:07:46.191830 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:07:46.192102 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:07:46.192105 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:07:46.192106 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:07:46.192179 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:07:46.192580 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:07:46.192585 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:07:46.192798 (+ 213us) client_negotiation.cc:769] Sending connection context
0903 08:07:46.192964 (+ 166us) client_negotiation.cc:240] Negotiation successful
0903 08:07:46.193176 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":103,"threads_started":1}
WARNING: no policy specified for kudu/127.12.201.195@KRBTEST.COM; defaulting to no policy
I20250903 08:07:46.194943 14006 heartbeater.cc:344] Connected to a master server at 127.12.201.254:40795
I20250903 08:07:46.195230 14006 heartbeater.cc:461] Registering TS with master...
I20250903 08:07:46.195752 14006 heartbeater.cc:507] Master 127.12.201.254:40795 requested a full tablet report, sending...
I20250903 08:07:46.196841 13677 ts_manager.cc:194] Registered new tserver with Master: 23477b2f0bc846249af230723bdea921 (127.12.201.194:33221)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.201.195@KRBTEST.COM" created.
I20250903 08:07:46.197391 13677 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.201.194@KRBTEST.COM'} at 127.12.201.194:48255
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.201.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.201.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.201.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.201.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.201.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.201.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/krb5kdc/kudu.keytab.
I20250903 08:07:46.229910 13095 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUML90r/build/debug/bin/kudu
/tmp/dist-test-taskUML90r/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.201.195:0
--local_ip_for_outbound_sockets=127.12.201.195
--webserver_interface=127.12.201.195
--webserver_port=0
--tserver_master_addrs=127.12.201.254:40795
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.201.212:39179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 08:07:46.335621 14013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:07:46.335821 14013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:07:46.335897 14013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:07:46.339038 14013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 08:07:46.339093 14013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:07:46.339198 14013 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.201.195
I20250903 08:07:46.343245 14013 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.201.212:39179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.201.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.201.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.12.201.254:40795
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14013
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.201.195
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 07:50:08 UTC on 5fd53c4cbb9d
build id 7874
I20250903 08:07:46.344208 14013 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:07:46.345021 14013 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:07:46.351274 14021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:07:46.351346 14018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:07:46.351473 14013 server_base.cc:1047] running on GCE node
W20250903 08:07:46.351277 14019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:07:46.351841 14013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:07:46.352348 14013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:07:46.353497 14013 hybrid_clock.cc:648] HybridClock initialized: now 1756886866353486 us; error 26 us; skew 500 ppm
Sep 03 08:07:46 dist-test-slave-27r9 krb5kdc[13629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886866, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 08:07:46.356278 14013 init.cc:377] Logged in from keytab as kudu/127.12.201.195@KRBTEST.COM (short username kudu)
I20250903 08:07:46.357268 14013 webserver.cc:480] Webserver started at http://127.12.201.195:40985/ using document root <none> and password file <none>
I20250903 08:07:46.357797 14013 fs_manager.cc:362] Metadata directory not provided
I20250903 08:07:46.357851 14013 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:07:46.358024 14013 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:07:46.359753 14013 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/minicluster-data/ts-2/data/instance:
uuid: "b439bca365cb454ea0826ab3224bc215"
format_stamp: "Formatted at 2025-09-03 08:07:46 on dist-test-slave-27r9"
server_key: "c579fba3b181c69b49859b409c3ced58"
server_key_iv: "50a257ec060531ba77a17a9ba4841d61"
server_key_version: "encryptionkey@0"
I20250903 08:07:46.360220 14013 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/minicluster-data/ts-2/wal/instance:
uuid: "b439bca365cb454ea0826ab3224bc215"
format_stamp: "Formatted at 2025-09-03 08:07:46 on dist-test-slave-27r9"
server_key: "c579fba3b181c69b49859b409c3ced58"
server_key_iv: "50a257ec060531ba77a17a9ba4841d61"
server_key_version: "encryptionkey@0"
I20250903 08:07:46.363503 14013 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250903 08:07:46.365546 14028 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:07:46.366477 14013 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250903 08:07:46.366585 14013 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/minicluster-data/ts-2/wal
uuid: "b439bca365cb454ea0826ab3224bc215"
format_stamp: "Formatted at 2025-09-03 08:07:46 on dist-test-slave-27r9"
server_key: "c579fba3b181c69b49859b409c3ced58"
server_key_iv: "50a257ec060531ba77a17a9ba4841d61"
server_key_version: "encryptionkey@0"
I20250903 08:07:46.366664 14013 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:07:46.383827 14013 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:07:46.386310 14013 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:07:46.386490 14013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:07:46.387044 14013 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 08:07:46.387866 14013 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 08:07:46.387915 14013 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 08:07:46.387974 14013 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 08:07:46.388006 14013 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 08:07:46.396205 14013 rpc_server.cc:307] RPC server started. Bound to: 127.12.201.195:45587
I20250903 08:07:46.396230 14141 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.201.195:45587 every 8 connection(s)
I20250903 08:07:46.397123 14013 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/minicluster-data/ts-2/data/info.pb
Sep 03 08:07:46 dist-test-slave-27r9 krb5kdc[13629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886866, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.195@KRBTEST.COM for kudu/127.12.201.254@KRBTEST.COM
I20250903 08:07:46.405244 13095 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUML90r/build/debug/bin/kudu as pid 14013
I20250903 08:07:46.405357 13095 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756886860253814-13095-0/minicluster-data/ts-2/wal/instance
I20250903 08:07:46.405587 13095 external_mini_cluster.cc:1468] Setting key ef53d1899babecb163afb16ab616c772
I20250903 08:07:46.408914 13716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:46.398851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.195:59829 (local address 127.12.201.254:40795)
0903 08:07:46.399047 (+ 196us) server_negotiation.cc:207] Beginning negotiation
0903 08:07:46.399051 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:07:46.399686 (+ 635us) server_negotiation.cc:408] Connection header received
0903 08:07:46.400441 (+ 755us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:07:46.400444 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:07:46.400491 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:07:46.400565 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:07:46.401762 (+ 1197us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:46.402245 (+ 483us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:07:46.403346 (+ 1101us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:46.403479 (+ 133us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:46.405599 (+ 2120us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:07:46.405623 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:07:46.405628 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:07:46.405660 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:07:46.407105 (+ 1445us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:07:46.407567 (+ 462us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:07:46.407573 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:07:46.407575 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:07:46.407618 (+ 43us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:07:46.407872 (+ 254us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:07:46.407875 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:07:46.407877 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:07:46.408021 (+ 144us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:07:46.408113 (+ 92us) server_negotiation.cc:1036] Waiting for connection context
0903 08:07:46.408637 (+ 524us) server_negotiation.cc:300] Negotiation successful
0903 08:07:46.408753 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20250903 08:07:46.409539 14144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:46.399130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:40795 (local address 127.12.201.195:59829)
0903 08:07:46.399542 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0903 08:07:46.399579 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0903 08:07:46.400268 (+ 689us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:07:46.400697 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:07:46.400704 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:07:46.401083 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:07:46.401630 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:46.401643 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:46.402331 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:07:46.402334 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:07:46.403248 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:46.403257 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:46.403419 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:46.403975 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:07:46.403997 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:07:46.405441 (+ 1444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:07:46.407211 (+ 1770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:07:46.407217 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:07:46.407228 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:07:46.407462 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:07:46.407705 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:07:46.407710 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:07:46.407712 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:07:46.407789 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:07:46.408169 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:07:46.408188 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:07:46.408401 (+ 213us) client_negotiation.cc:769] Sending connection context
0903 08:07:46.408585 (+ 184us) client_negotiation.cc:240] Negotiation successful
0903 08:07:46.408840 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":102,"threads_started":1}
I20250903 08:07:46.410434 14142 heartbeater.cc:344] Connected to a master server at 127.12.201.254:40795
I20250903 08:07:46.410636 14142 heartbeater.cc:461] Registering TS with master...
I20250903 08:07:46.411119 14142 heartbeater.cc:507] Master 127.12.201.254:40795 requested a full tablet report, sending...
I20250903 08:07:46.412009 13677 ts_manager.cc:194] Registered new tserver with Master: b439bca365cb454ea0826ab3224bc215 (127.12.201.195:45587)
I20250903 08:07:46.412544 13677 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.201.195@KRBTEST.COM'} at 127.12.201.195:59829
I20250903 08:07:46.418848 13095 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250903 08:07:46.427661 13716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:46.421539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39368 (local address 127.12.201.254:40795)
0903 08:07:46.421683 (+ 144us) server_negotiation.cc:207] Beginning negotiation
0903 08:07:46.421689 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0903 08:07:46.421795 (+ 106us) server_negotiation.cc:408] Connection header received
0903 08:07:46.421906 (+ 111us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:07:46.421908 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:07:46.421948 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:07:46.422013 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:07:46.422765 (+ 752us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:46.423274 (+ 509us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:07:46.423953 (+ 679us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:46.424095 (+ 142us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:46.425145 (+ 1050us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:07:46.425159 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:07:46.425162 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:07:46.425184 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:07:46.426383 (+ 1199us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:07:46.426721 (+ 338us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:07:46.426723 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:07:46.426724 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:07:46.426757 (+ 33us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:07:46.426906 (+ 149us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:07:46.426908 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:07:46.426909 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:07:46.427098 (+ 189us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:07:46.427201 (+ 103us) server_negotiation.cc:1036] Waiting for connection context
0903 08:07:46.427407 (+ 206us) server_negotiation.cc:300] Negotiation successful
0903 08:07:46.427507 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20250903 08:07:46.431447 13677 catalog_manager.cc:2232] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39368:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 08:07:46.433862 13677 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 08:07:46.446197 14153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:46.441586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.193:41851 (local address 127.0.0.1:59526)
0903 08:07:46.441936 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0903 08:07:46.441977 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0903 08:07:46.442114 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:07:46.443324 (+ 1210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:07:46.443329 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:07:46.443353 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 08:07:46.443685 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:46.443695 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:46.445085 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:07:46.445088 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:07:46.445882 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:46.445889 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:46.445978 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:46.445990 (+ 12us) client_negotiation.cc:769] Sending connection context
0903 08:07:46.446034 (+ 44us) client_negotiation.cc:240] Negotiation successful
0903 08:07:46.446075 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":105,"threads_started":1}
I20250903 08:07:46.446197 14154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:46.441620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.194:33221 (local address 127.0.0.1:43846)
0903 08:07:46.442410 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0903 08:07:46.442424 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0903 08:07:46.442867 (+ 443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:07:46.443465 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:07:46.443468 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:07:46.443484 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 08:07:46.443732 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:46.443740 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:46.444789 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:07:46.444793 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:07:46.445632 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:46.445641 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:46.445796 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:46.445849 (+ 53us) client_negotiation.cc:769] Sending connection context
0903 08:07:46.445942 (+ 93us) client_negotiation.cc:240] Negotiation successful
0903 08:07:46.446038 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":713,"thread_start_us":88,"threads_started":1}
I20250903 08:07:46.446836 14156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:46.442281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.195:45587 (local address 127.0.0.1:54416)
0903 08:07:46.442709 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0903 08:07:46.442723 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0903 08:07:46.442806 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:07:46.443624 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:07:46.443627 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:07:46.443640 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 08:07:46.443833 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:46.443839 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:46.445134 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:07:46.445138 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:07:46.445966 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:46.445975 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:46.446081 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:46.446103 (+ 22us) client_negotiation.cc:769] Sending connection context
0903 08:07:46.446155 (+ 52us) client_negotiation.cc:240] Negotiation successful
0903 08:07:46.446211 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":122,"threads_started":1}
I20250903 08:07:46.447126 14158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:46.442425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54416 (local address 127.12.201.195:45587)
0903 08:07:46.443138 (+ 713us) server_negotiation.cc:207] Beginning negotiation
0903 08:07:46.443145 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0903 08:07:46.443166 (+ 21us) server_negotiation.cc:408] Connection header received
0903 08:07:46.443320 (+ 154us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:07:46.443326 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:07:46.443482 (+ 156us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:07:46.443625 (+ 143us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 08:07:46.443943 (+ 318us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:46.445014 (+ 1071us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:07:46.446115 (+ 1101us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:46.446659 (+ 544us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:46.446784 (+ 125us) server_negotiation.cc:1036] Waiting for connection context
0903 08:07:46.446848 (+ 64us) server_negotiation.cc:300] Negotiation successful
0903 08:07:46.446944 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":135,"threads_started":1}
I20250903 08:07:46.447304 14157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:46.442368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43846 (local address 127.12.201.194:33221)
0903 08:07:46.443045 (+ 677us) server_negotiation.cc:207] Beginning negotiation
0903 08:07:46.443051 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0903 08:07:46.443069 (+ 18us) server_negotiation.cc:408] Connection header received
0903 08:07:46.443146 (+ 77us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:07:46.443150 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:07:46.443314 (+ 164us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:07:46.443869 (+ 555us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 08:07:46.443994 (+ 125us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:46.444682 (+ 688us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:07:46.446276 (+ 1594us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:46.446870 (+ 594us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:46.447021 (+ 151us) server_negotiation.cc:1036] Waiting for connection context
0903 08:07:46.447085 (+ 64us) server_negotiation.cc:300] Negotiation successful
0903 08:07:46.447153 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":120,"threads_started":1}
I20250903 08:07:46.447984 14155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:46.441685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59526 (local address 127.12.201.193:41851)
0903 08:07:46.442728 (+ 1043us) server_negotiation.cc:207] Beginning negotiation
0903 08:07:46.442737 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0903 08:07:46.442763 (+ 26us) server_negotiation.cc:408] Connection header received
0903 08:07:46.442852 (+ 89us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:07:46.442858 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:07:46.443121 (+ 263us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:07:46.443311 (+ 190us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 08:07:46.443820 (+ 509us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:46.444969 (+ 1149us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:07:46.447085 (+ 2116us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:46.447611 (+ 526us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:46.447701 (+ 90us) server_negotiation.cc:1036] Waiting for connection context
0903 08:07:46.447765 (+ 64us) server_negotiation.cc:300] Negotiation successful
0903 08:07:46.447845 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":942,"thread_start_us":112,"threads_started":1}
I20250903 08:07:46.449823 14076 tablet_service.cc:1468] Processing CreateTablet for tablet 09b7c479b8f44953b28277f378d324a5 (DEFAULT_TABLE table=test-table [id=6bb7ad2bb78e4588b882d05d888da4b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:07:46.449828 13940 tablet_service.cc:1468] Processing CreateTablet for tablet 09b7c479b8f44953b28277f378d324a5 (DEFAULT_TABLE table=test-table [id=6bb7ad2bb78e4588b882d05d888da4b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:07:46.450044 13804 tablet_service.cc:1468] Processing CreateTablet for tablet 09b7c479b8f44953b28277f378d324a5 (DEFAULT_TABLE table=test-table [id=6bb7ad2bb78e4588b882d05d888da4b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:07:46.450814 13804 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09b7c479b8f44953b28277f378d324a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:07:46.450821 14076 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09b7c479b8f44953b28277f378d324a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:07:46.450861 13940 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09b7c479b8f44953b28277f378d324a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:07:46.456573 14159 tablet_bootstrap.cc:492] T 09b7c479b8f44953b28277f378d324a5 P 6209003669834836875868588d559ce9: Bootstrap starting.
I20250903 08:07:46.457815 14160 tablet_bootstrap.cc:492] T 09b7c479b8f44953b28277f378d324a5 P b439bca365cb454ea0826ab3224bc215: Bootstrap starting.
I20250903 08:07:46.458281 14161 tablet_bootstrap.cc:492] T 09b7c479b8f44953b28277f378d324a5 P 23477b2f0bc846249af230723bdea921: Bootstrap starting.
I20250903 08:07:46.459467 14159 tablet_bootstrap.cc:654] T 09b7c479b8f44953b28277f378d324a5 P 6209003669834836875868588d559ce9: Neither blocks nor log segments found. Creating new log.
I20250903 08:07:46.459846 14160 tablet_bootstrap.cc:654] T 09b7c479b8f44953b28277f378d324a5 P b439bca365cb454ea0826ab3224bc215: Neither blocks nor log segments found. Creating new log.
I20250903 08:07:46.460207 14159 log.cc:826] T 09b7c479b8f44953b28277f378d324a5 P 6209003669834836875868588d559ce9: Log is configured to *not* fsync() on all Append() calls
I20250903 08:07:46.460394 14161 tablet_bootstrap.cc:654] T 09b7c479b8f44953b28277f378d324a5 P 23477b2f0bc846249af230723bdea921: Neither blocks nor log segments found. Creating new log.
I20250903 08:07:46.460654 14160 log.cc:826] T 09b7c479b8f44953b28277f378d324a5 P b439bca365cb454ea0826ab3224bc215: Log is configured to *not* fsync() on all Append() calls
I20250903 08:07:46.461124 14161 log.cc:826] T 09b7c479b8f44953b28277f378d324a5 P 23477b2f0bc846249af230723bdea921: Log is configured to *not* fsync() on all Append() calls
I20250903 08:07:46.462131 14159 tablet_bootstrap.cc:492] T 09b7c479b8f44953b28277f378d324a5 P 6209003669834836875868588d559ce9: No bootstrap required, opened a new log
I20250903 08:07:46.462316 14159 ts_tablet_manager.cc:1397] T 09b7c479b8f44953b28277f378d324a5 P 6209003669834836875868588d559ce9: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20250903 08:07:46.462394 14160 tablet_bootstrap.cc:492] T 09b7c479b8f44953b28277f378d324a5 P b439bca365cb454ea0826ab3224bc215: No bootstrap required, opened a new log
I20250903 08:07:46.462569 14160 ts_tablet_manager.cc:1397] T 09b7c479b8f44953b28277f378d324a5 P b439bca365cb454ea0826ab3224bc215: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20250903 08:07:46.462889 14161 tablet_bootstrap.cc:492] T 09b7c479b8f44953b28277f378d324a5 P 23477b2f0bc846249af230723bdea921: No bootstrap required, opened a new log
I20250903 08:07:46.463083 14161 ts_tablet_manager.cc:1397] T 09b7c479b8f44953b28277f378d324a5 P 23477b2f0bc846249af230723bdea921: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20250903 08:07:46.464987 14159 raft_consensus.cc:357] T 09b7c479b8f44953b28277f378d324a5 P 6209003669834836875868588d559ce9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6209003669834836875868588d559ce9" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 41851 } } peers { permanent_uuid: "23477b2f0bc846249af230723bdea921" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 33221 } } peers { permanent_uuid: "b439bca365cb454ea0826ab3224bc215" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 45587 } }
I20250903 08:07:46.465171 14159 raft_consensus.cc:383] T 09b7c479b8f44953b28277f378d324a5 P 6209003669834836875868588d559ce9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:07:46.465235 14159 raft_consensus.cc:738] T 09b7c479b8f44953b28277f378d324a5 P 6209003669834836875868588d559ce9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6209003669834836875868588d559ce9, State: Initialized, Role: FOLLOWER
I20250903 08:07:46.465637 14159 consensus_queue.cc:260] T 09b7c479b8f44953b28277f378d324a5 P 6209003669834836875868588d559ce9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6209003669834836875868588d559ce9" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 41851 } } peers { permanent_uuid: "23477b2f0bc846249af230723bdea921" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 33221 } } peers { permanent_uuid: "b439bca365cb454ea0826ab3224bc215" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 45587 } }
I20250903 08:07:46.465735 14160 raft_consensus.cc:357] T 09b7c479b8f44953b28277f378d324a5 P b439bca365cb454ea0826ab3224bc215 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6209003669834836875868588d559ce9" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 41851 } } peers { permanent_uuid: "23477b2f0bc846249af230723bdea921" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 33221 } } peers { permanent_uuid: "b439bca365cb454ea0826ab3224bc215" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 45587 } }
I20250903 08:07:46.465965 14160 raft_consensus.cc:383] T 09b7c479b8f44953b28277f378d324a5 P b439bca365cb454ea0826ab3224bc215 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:07:46.466037 14160 raft_consensus.cc:738] T 09b7c479b8f44953b28277f378d324a5 P b439bca365cb454ea0826ab3224bc215 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b439bca365cb454ea0826ab3224bc215, State: Initialized, Role: FOLLOWER
I20250903 08:07:46.466398 13870 heartbeater.cc:499] Master 127.12.201.254:40795 was elected leader, sending a full tablet report...
I20250903 08:07:46.466501 14160 consensus_queue.cc:260] T 09b7c479b8f44953b28277f378d324a5 P b439bca365cb454ea0826ab3224bc215 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6209003669834836875868588d559ce9" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 41851 } } peers { permanent_uuid: "23477b2f0bc846249af230723bdea921" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 33221 } } peers { permanent_uuid: "b439bca365cb454ea0826ab3224bc215" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 45587 } }
I20250903 08:07:46.466730 14159 ts_tablet_manager.cc:1428] T 09b7c479b8f44953b28277f378d324a5 P 6209003669834836875868588d559ce9: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20250903 08:07:46.466111 14161 raft_consensus.cc:357] T 09b7c479b8f44953b28277f378d324a5 P 23477b2f0bc846249af230723bdea921 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6209003669834836875868588d559ce9" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 41851 } } peers { permanent_uuid: "23477b2f0bc846249af230723bdea921" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 33221 } } peers { permanent_uuid: "b439bca365cb454ea0826ab3224bc215" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 45587 } }
I20250903 08:07:46.466887 14161 raft_consensus.cc:383] T 09b7c479b8f44953b28277f378d324a5 P 23477b2f0bc846249af230723bdea921 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:07:46.466948 14161 raft_consensus.cc:738] T 09b7c479b8f44953b28277f378d324a5 P 23477b2f0bc846249af230723bdea921 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23477b2f0bc846249af230723bdea921, State: Initialized, Role: FOLLOWER
I20250903 08:07:46.467331 14160 ts_tablet_manager.cc:1428] T 09b7c479b8f44953b28277f378d324a5 P b439bca365cb454ea0826ab3224bc215: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20250903 08:07:46.467374 14161 consensus_queue.cc:260] T 09b7c479b8f44953b28277f378d324a5 P 23477b2f0bc846249af230723bdea921 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6209003669834836875868588d559ce9" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 41851 } } peers { permanent_uuid: "23477b2f0bc846249af230723bdea921" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 33221 } } peers { permanent_uuid: "b439bca365cb454ea0826ab3224bc215" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 45587 } }
I20250903 08:07:46.467506 14142 heartbeater.cc:499] Master 127.12.201.254:40795 was elected leader, sending a full tablet report...
I20250903 08:07:46.467906 14006 heartbeater.cc:499] Master 127.12.201.254:40795 was elected leader, sending a full tablet report...
I20250903 08:07:46.468113 14161 ts_tablet_manager.cc:1428] T 09b7c479b8f44953b28277f378d324a5 P 23477b2f0bc846249af230723bdea921: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20250903 08:07:46.584800 14167 raft_consensus.cc:491] T 09b7c479b8f44953b28277f378d324a5 P 23477b2f0bc846249af230723bdea921 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 08:07:46.585049 14167 raft_consensus.cc:513] T 09b7c479b8f44953b28277f378d324a5 P 23477b2f0bc846249af230723bdea921 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6209003669834836875868588d559ce9" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 41851 } } peers { permanent_uuid: "23477b2f0bc846249af230723bdea921" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 33221 } } peers { permanent_uuid: "b439bca365cb454ea0826ab3224bc215" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 45587 } }
I20250903 08:07:46.586256 14167 leader_election.cc:290] T 09b7c479b8f44953b28277f378d324a5 P 23477b2f0bc846249af230723bdea921 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6209003669834836875868588d559ce9 (127.12.201.193:41851), b439bca365cb454ea0826ab3224bc215 (127.12.201.195:45587)
I20250903 08:07:46.589569 14008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:46.586551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.193:41851 (local address 127.12.201.194:59555)
0903 08:07:46.586808 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0903 08:07:46.586832 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0903 08:07:46.586936 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:07:46.587332 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:07:46.587334 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:07:46.587347 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 08:07:46.587621 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:46.587631 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:46.588476 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:07:46.588479 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:07:46.589169 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:46.589177 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:46.589287 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:46.589311 (+ 24us) client_negotiation.cc:769] Sending connection context
0903 08:07:46.589363 (+ 52us) client_negotiation.cc:240] Negotiation successful
0903 08:07:46.589420 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141}
I20250903 08:07:46.589973 14155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:46.586720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.194:59555 (local address 127.12.201.193:41851)
0903 08:07:46.586873 (+ 153us) server_negotiation.cc:207] Beginning negotiation
0903 08:07:46.586877 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:07:46.586897 (+ 20us) server_negotiation.cc:408] Connection header received
0903 08:07:46.587118 (+ 221us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:07:46.587122 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:07:46.587219 (+ 97us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:07:46.587312 (+ 93us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 08:07:46.587774 (+ 462us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:46.588356 (+ 582us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:07:46.589304 (+ 948us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:46.589679 (+ 375us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:46.589732 (+ 53us) server_negotiation.cc:1036] Waiting for connection context
0903 08:07:46.589784 (+ 52us) server_negotiation.cc:300] Negotiation successful
0903 08:07:46.589842 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20250903 08:07:46.590165 14168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:46.586592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.195:45587 (local address 127.12.201.194:36389)
0903 08:07:46.587071 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0903 08:07:46.587085 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0903 08:07:46.587158 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:07:46.587409 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:07:46.587413 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:07:46.587431 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 08:07:46.588192 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:46.588198 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:46.588983 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:07:46.588987 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:07:46.589824 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:46.589835 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:46.589920 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:46.589932 (+ 12us) client_negotiation.cc:769] Sending connection context
0903 08:07:46.589967 (+ 35us) client_negotiation.cc:240] Negotiation successful
0903 08:07:46.590010 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":145,"threads_started":1}
I20250903 08:07:46.590611 13824 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "09b7c479b8f44953b28277f378d324a5" candidate_uuid: "23477b2f0bc846249af230723bdea921" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6209003669834836875868588d559ce9" is_pre_election: true
I20250903 08:07:46.590719 14158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:46.586710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.194:36389 (local address 127.12.201.195:45587)
0903 08:07:46.586839 (+ 129us) server_negotiation.cc:207] Beginning negotiation
0903 08:07:46.586845 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0903 08:07:46.587160 (+ 315us) server_negotiation.cc:408] Connection header received
0903 08:07:46.587258 (+ 98us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:07:46.587264 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:07:46.587321 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:07:46.587390 (+ 69us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 08:07:46.588320 (+ 930us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:46.588860 (+ 540us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:07:46.589945 (+ 1085us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:46.590447 (+ 502us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:46.590482 (+ 35us) server_negotiation.cc:1036] Waiting for connection context
0903 08:07:46.590546 (+ 64us) server_negotiation.cc:300] Negotiation successful
0903 08:07:46.590604 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20250903 08:07:46.590942 13824 raft_consensus.cc:2466] T 09b7c479b8f44953b28277f378d324a5 P 6209003669834836875868588d559ce9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23477b2f0bc846249af230723bdea921 in term 0.
I20250903 08:07:46.591320 14096 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "09b7c479b8f44953b28277f378d324a5" candidate_uuid: "23477b2f0bc846249af230723bdea921" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b439bca365cb454ea0826ab3224bc215" is_pre_election: true
I20250903 08:07:46.591488 13893 leader_election.cc:304] T 09b7c479b8f44953b28277f378d324a5 P 23477b2f0bc846249af230723bdea921 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23477b2f0bc846249af230723bdea921, 6209003669834836875868588d559ce9; no voters:
I20250903 08:07:46.591594 14096 raft_consensus.cc:2466] T 09b7c479b8f44953b28277f378d324a5 P b439bca365cb454ea0826ab3224bc215 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23477b2f0bc846249af230723bdea921 in term 0.
I20250903 08:07:46.591782 14167 raft_consensus.cc:2802] T 09b7c479b8f44953b28277f378d324a5 P 23477b2f0bc846249af230723bdea921 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 08:07:46.591863 14167 raft_consensus.cc:491] T 09b7c479b8f44953b28277f378d324a5 P 23477b2f0bc846249af230723bdea921 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 08:07:46.591923 14167 raft_consensus.cc:3058] T 09b7c479b8f44953b28277f378d324a5 P 23477b2f0bc846249af230723bdea921 [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:07:46.592895 14167 raft_consensus.cc:513] T 09b7c479b8f44953b28277f378d324a5 P 23477b2f0bc846249af230723bdea921 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6209003669834836875868588d559ce9" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 41851 } } peers { permanent_uuid: "23477b2f0bc846249af230723bdea921" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 33221 } } peers { permanent_uuid: "b439bca365cb454ea0826ab3224bc215" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 45587 } }
I20250903 08:07:46.593220 14167 leader_election.cc:290] T 09b7c479b8f44953b28277f378d324a5 P 23477b2f0bc846249af230723bdea921 [CANDIDATE]: Term 1 election: Requested vote from peers 6209003669834836875868588d559ce9 (127.12.201.193:41851), b439bca365cb454ea0826ab3224bc215 (127.12.201.195:45587)
I20250903 08:07:46.593607 14096 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "09b7c479b8f44953b28277f378d324a5" candidate_uuid: "23477b2f0bc846249af230723bdea921" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b439bca365cb454ea0826ab3224bc215"
I20250903 08:07:46.593760 14096 raft_consensus.cc:3058] T 09b7c479b8f44953b28277f378d324a5 P b439bca365cb454ea0826ab3224bc215 [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:07:46.593739 13824 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "09b7c479b8f44953b28277f378d324a5" candidate_uuid: "23477b2f0bc846249af230723bdea921" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6209003669834836875868588d559ce9"
I20250903 08:07:46.593873 13824 raft_consensus.cc:3058] T 09b7c479b8f44953b28277f378d324a5 P 6209003669834836875868588d559ce9 [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:07:46.594864 14096 raft_consensus.cc:2466] T 09b7c479b8f44953b28277f378d324a5 P b439bca365cb454ea0826ab3224bc215 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23477b2f0bc846249af230723bdea921 in term 1.
I20250903 08:07:46.594929 13824 raft_consensus.cc:2466] T 09b7c479b8f44953b28277f378d324a5 P 6209003669834836875868588d559ce9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23477b2f0bc846249af230723bdea921 in term 1.
I20250903 08:07:46.595232 13896 leader_election.cc:304] T 09b7c479b8f44953b28277f378d324a5 P 23477b2f0bc846249af230723bdea921 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23477b2f0bc846249af230723bdea921, b439bca365cb454ea0826ab3224bc215; no voters:
I20250903 08:07:46.595467 14167 raft_consensus.cc:2802] T 09b7c479b8f44953b28277f378d324a5 P 23477b2f0bc846249af230723bdea921 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 08:07:46.595705 14167 raft_consensus.cc:695] T 09b7c479b8f44953b28277f378d324a5 P 23477b2f0bc846249af230723bdea921 [term 1 LEADER]: Becoming Leader. State: Replica: 23477b2f0bc846249af230723bdea921, State: Running, Role: LEADER
I20250903 08:07:46.596043 14167 consensus_queue.cc:237] T 09b7c479b8f44953b28277f378d324a5 P 23477b2f0bc846249af230723bdea921 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6209003669834836875868588d559ce9" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 41851 } } peers { permanent_uuid: "23477b2f0bc846249af230723bdea921" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 33221 } } peers { permanent_uuid: "b439bca365cb454ea0826ab3224bc215" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 45587 } }
I20250903 08:07:46.599442 13677 catalog_manager.cc:5582] T 09b7c479b8f44953b28277f378d324a5 P 23477b2f0bc846249af230723bdea921 reported cstate change: term changed from 0 to 1, leader changed from <none> to 23477b2f0bc846249af230723bdea921 (127.12.201.194). New cstate: current_term: 1 leader_uuid: "23477b2f0bc846249af230723bdea921" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6209003669834836875868588d559ce9" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 41851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23477b2f0bc846249af230723bdea921" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 33221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b439bca365cb454ea0826ab3224bc215" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 45587 } health_report { overall_health: UNKNOWN } } }
Sep 03 08:07:46 dist-test-slave-27r9 krb5kdc[13629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886865, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.201.193@KRBTEST.COM
I20250903 08:07:46.638680 14155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:46.630394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59532 (local address 127.12.201.193:41851)
0903 08:07:46.630521 (+ 127us) server_negotiation.cc:207] Beginning negotiation
0903 08:07:46.630525 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:07:46.630612 (+ 87us) server_negotiation.cc:408] Connection header received
0903 08:07:46.630712 (+ 100us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:07:46.630715 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:07:46.630761 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:07:46.630833 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:07:46.631704 (+ 871us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:46.632170 (+ 466us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:07:46.632825 (+ 655us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:46.632949 (+ 124us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:46.635471 (+ 2522us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:07:46.635500 (+ 29us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:07:46.635514 (+ 14us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:07:46.635544 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:07:46.637321 (+ 1777us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:07:46.637722 (+ 401us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:07:46.637727 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:07:46.637728 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:07:46.637781 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:07:46.637938 (+ 157us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:07:46.637940 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:07:46.637942 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:07:46.638125 (+ 183us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:07:46.638280 (+ 155us) server_negotiation.cc:1036] Waiting for connection context
0903 08:07:46.638439 (+ 159us) server_negotiation.cc:300] Negotiation successful
0903 08:07:46.638548 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20250903 08:07:46.647979 14143 tablet.cc:2378] T 09b7c479b8f44953b28277f378d324a5 P b439bca365cb454ea0826ab3224bc215: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 03 08:07:46 dist-test-slave-27r9 krb5kdc[13629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886866, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 03 08:07:46 dist-test-slave-27r9 krb5kdc[13629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886866, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.201.193@KRBTEST.COM
I20250903 08:07:46.663774 14155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:46.655795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59540 (local address 127.12.201.193:41851)
0903 08:07:46.655950 (+ 155us) server_negotiation.cc:207] Beginning negotiation
0903 08:07:46.655956 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0903 08:07:46.656037 (+ 81us) server_negotiation.cc:408] Connection header received
0903 08:07:46.656195 (+ 158us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:07:46.656198 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:07:46.656258 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:07:46.656331 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:07:46.657205 (+ 874us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:46.657730 (+ 525us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:07:46.658376 (+ 646us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:46.658592 (+ 216us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:46.660765 (+ 2173us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:07:46.660787 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:07:46.660789 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:07:46.660820 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:07:46.662288 (+ 1468us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:07:46.662685 (+ 397us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:07:46.662689 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:07:46.662690 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:07:46.662735 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:07:46.662943 (+ 208us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:07:46.662947 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:07:46.662949 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:07:46.663131 (+ 182us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:07:46.663259 (+ 128us) server_negotiation.cc:1036] Waiting for connection context
0903 08:07:46.663490 (+ 231us) server_negotiation.cc:300] Negotiation successful
0903 08:07:46.663612 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Sep 03 08:07:46 dist-test-slave-27r9 krb5kdc[13629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886866, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
W20250903 08:07:46.681406 14007 tablet.cc:2378] T 09b7c479b8f44953b28277f378d324a5 P 23477b2f0bc846249af230723bdea921: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 03 08:07:46 dist-test-slave-27r9 krb5kdc[13629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886866, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.201.193@KRBTEST.COM
I20250903 08:07:46.687711 14155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:46.680277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59546 (local address 127.12.201.193:41851)
0903 08:07:46.680420 (+ 143us) server_negotiation.cc:207] Beginning negotiation
0903 08:07:46.680426 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0903 08:07:46.680524 (+ 98us) server_negotiation.cc:408] Connection header received
0903 08:07:46.680680 (+ 156us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:07:46.680683 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:07:46.680730 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:07:46.680804 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:07:46.681740 (+ 936us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:46.682203 (+ 463us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:07:46.682810 (+ 607us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:46.682946 (+ 136us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:46.685016 (+ 2070us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:07:46.685040 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:07:46.685042 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:07:46.685068 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:07:46.686319 (+ 1251us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:07:46.686668 (+ 349us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:07:46.686670 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:07:46.686672 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:07:46.686705 (+ 33us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:07:46.686851 (+ 146us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:07:46.686852 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:07:46.686854 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:07:46.686966 (+ 112us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:07:46.687130 (+ 164us) server_negotiation.cc:1036] Waiting for connection context
0903 08:07:46.687430 (+ 300us) server_negotiation.cc:300] Negotiation successful
0903 08:07:46.687549 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20250903 08:07:46.688954 13784 tablet_service.cc:2912] Rejecting scan request for tablet 09b7c479b8f44953b28277f378d324a5: Uninitialized: clean time has not yet been initialized
I20250903 08:07:46.690829 13095 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUML90r/build/debug/bin/kudu with pid 13741
I20250903 08:07:46.697120 13095 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUML90r/build/debug/bin/kudu with pid 13877
I20250903 08:07:46.702759 13095 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUML90r/build/debug/bin/kudu with pid 14013
I20250903 08:07:46.708410 13095 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUML90r/build/debug/bin/kudu with pid 13645
2025-09-03T08:07:46Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (3313 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 03 08:07:46 dist-test-slave-27r9 krb5kdc[14183](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 03 08:07:46 dist-test-slave-27r9 krb5kdc[14183](info): set up 2 sockets
Sep 03 08:07:46 dist-test-slave-27r9 krb5kdc[14183](info): commencing operation
krb5kdc: starting...
W20250903 08:07:48.707325 13095 mini_kdc.cc:121] Time spent starting KDC: real 1.975s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/krb5kdc/test-user.keytab.
Sep 03 08:07:48 dist-test-slave-27r9 krb5kdc[14183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886868, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-09-03T08:07:48Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-03T08:07:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.201.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.201.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.201.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.201.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.201.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.201.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.201.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.201.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/krb5kdc/kudu.keytab.
I20250903 08:07:48.840129 13095 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUML90r/build/debug/bin/kudu
/tmp/dist-test-taskUML90r/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.201.254:46669
--webserver_interface=127.12.201.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.201.212:37769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.201.254:46669
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 08:07:48.947629 14199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:07:48.947911 14199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:07:48.948011 14199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:07:48.951373 14199 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250903 08:07:48.951442 14199 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20250903 08:07:48.951460 14199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 08:07:48.951488 14199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:07:48.951510 14199 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250903 08:07:48.951550 14199 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250903 08:07:48.956058 14199 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.201.212:37769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.201.254:46669
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.201.254:46669
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.201.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 07:50:08 UTC on 5fd53c4cbb9d
build id 7874
I20250903 08:07:48.957191 14199 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:07:48.958087 14199 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:07:48.963826 14205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:07:48.963865 14204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:07:48.963984 14207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:07:48.964155 14199 server_base.cc:1047] running on GCE node
I20250903 08:07:48.964672 14199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:07:48.965582 14199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:07:48.966734 14199 hybrid_clock.cc:648] HybridClock initialized: now 1756886868966721 us; error 31 us; skew 500 ppm
Sep 03 08:07:48 dist-test-slave-27r9 krb5kdc[14183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886868, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 08:07:48.969393 14199 init.cc:377] Logged in from keytab as kudu/127.12.201.254@KRBTEST.COM (short username kudu)
I20250903 08:07:48.970446 14199 webserver.cc:480] Webserver started at http://127.12.201.254:32887/ using document root <none> and password file <none>
I20250903 08:07:48.971030 14199 fs_manager.cc:362] Metadata directory not provided
I20250903 08:07:48.971089 14199 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:07:48.971278 14199 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:07:48.972990 14199 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/minicluster-data/master-0/data/instance:
uuid: "5b996e3f77d84707b4bfc52734aeb4c9"
format_stamp: "Formatted at 2025-09-03 08:07:48 on dist-test-slave-27r9"
server_key: "31889b772393fb35b3894242c7a0c26d"
server_key_iv: "732fc7ef911467146aeb26a97a73a4b3"
server_key_version: "encryptionkey@0"
I20250903 08:07:48.973474 14199 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/minicluster-data/master-0/wal/instance:
uuid: "5b996e3f77d84707b4bfc52734aeb4c9"
format_stamp: "Formatted at 2025-09-03 08:07:48 on dist-test-slave-27r9"
server_key: "31889b772393fb35b3894242c7a0c26d"
server_key_iv: "732fc7ef911467146aeb26a97a73a4b3"
server_key_version: "encryptionkey@0"
I20250903 08:07:48.976917 14199 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250903 08:07:48.979205 14214 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:07:48.980190 14199 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250903 08:07:48.980315 14199 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/minicluster-data/master-0/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/minicluster-data/master-0/wal
uuid: "5b996e3f77d84707b4bfc52734aeb4c9"
format_stamp: "Formatted at 2025-09-03 08:07:48 on dist-test-slave-27r9"
server_key: "31889b772393fb35b3894242c7a0c26d"
server_key_iv: "732fc7ef911467146aeb26a97a73a4b3"
server_key_version: "encryptionkey@0"
I20250903 08:07:48.980409 14199 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:07:48.991935 14199 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:07:48.995110 14199 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:07:48.995298 14199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:07:49.003206 14199 rpc_server.cc:307] RPC server started. Bound to: 127.12.201.254:46669
I20250903 08:07:49.003213 14276 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.201.254:46669 every 8 connection(s)
I20250903 08:07:49.004208 14199 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/minicluster-data/master-0/data/info.pb
I20250903 08:07:49.005460 13095 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUML90r/build/debug/bin/kudu as pid 14199
I20250903 08:07:49.005539 13095 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/minicluster-data/master-0/wal/instance
I20250903 08:07:49.005739 13095 external_mini_cluster.cc:1468] Setting key 1ba2b15d09b9d11f99a36868ed8ae847
I20250903 08:07:49.006906 14277 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:07:49.013119 14277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b996e3f77d84707b4bfc52734aeb4c9: Bootstrap starting.
Sep 03 08:07:49 dist-test-slave-27r9 krb5kdc[14183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886868, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.201.254@KRBTEST.COM
I20250903 08:07:49.015630 14277 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b996e3f77d84707b4bfc52734aeb4c9: Neither blocks nor log segments found. Creating new log.
I20250903 08:07:49.016469 14277 log.cc:826] T 00000000000000000000000000000000 P 5b996e3f77d84707b4bfc52734aeb4c9: Log is configured to *not* fsync() on all Append() calls
I20250903 08:07:49.018215 14277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b996e3f77d84707b4bfc52734aeb4c9: No bootstrap required, opened a new log
I20250903 08:07:49.018544 14280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:49.006935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52322 (local address 127.12.201.254:46669)
0903 08:07:49.007355 (+ 420us) server_negotiation.cc:207] Beginning negotiation
0903 08:07:49.007366 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0903 08:07:49.007405 (+ 39us) server_negotiation.cc:408] Connection header received
0903 08:07:49.008330 (+ 925us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:07:49.008353 (+ 23us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:07:49.008698 (+ 345us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:07:49.009039 (+ 341us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:07:49.009775 (+ 736us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:49.011011 (+ 1236us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:07:49.011637 (+ 626us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:49.011904 (+ 267us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:49.013932 (+ 2028us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:07:49.013978 (+ 46us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:07:49.013991 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:07:49.014035 (+ 44us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:07:49.016171 (+ 2136us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:07:49.016576 (+ 405us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:07:49.016582 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:07:49.016587 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:07:49.016672 (+ 85us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:07:49.016896 (+ 224us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:07:49.016900 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:07:49.016902 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:07:49.017219 (+ 317us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:07:49.017401 (+ 182us) server_negotiation.cc:1036] Waiting for connection context
0903 08:07:49.017704 (+ 303us) server_negotiation.cc:300] Negotiation successful
0903 08:07:49.017940 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":114,"threads_started":1}
I20250903 08:07:49.021400 14277 raft_consensus.cc:357] T 00000000000000000000000000000000 P 5b996e3f77d84707b4bfc52734aeb4c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b996e3f77d84707b4bfc52734aeb4c9" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 46669 } }
I20250903 08:07:49.021579 14277 raft_consensus.cc:383] T 00000000000000000000000000000000 P 5b996e3f77d84707b4bfc52734aeb4c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:07:49.021646 14277 raft_consensus.cc:738] T 00000000000000000000000000000000 P 5b996e3f77d84707b4bfc52734aeb4c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b996e3f77d84707b4bfc52734aeb4c9, State: Initialized, Role: FOLLOWER
I20250903 08:07:49.022058 14277 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b996e3f77d84707b4bfc52734aeb4c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b996e3f77d84707b4bfc52734aeb4c9" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 46669 } }
I20250903 08:07:49.022184 14277 raft_consensus.cc:397] T 00000000000000000000000000000000 P 5b996e3f77d84707b4bfc52734aeb4c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250903 08:07:49.022249 14277 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5b996e3f77d84707b4bfc52734aeb4c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250903 08:07:49.022362 14277 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 5b996e3f77d84707b4bfc52734aeb4c9 [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:07:49.023406 14277 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5b996e3f77d84707b4bfc52734aeb4c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b996e3f77d84707b4bfc52734aeb4c9" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 46669 } }
I20250903 08:07:49.023699 14277 leader_election.cc:304] T 00000000000000000000000000000000 P 5b996e3f77d84707b4bfc52734aeb4c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b996e3f77d84707b4bfc52734aeb4c9; no voters:
I20250903 08:07:49.023931 14277 leader_election.cc:290] T 00000000000000000000000000000000 P 5b996e3f77d84707b4bfc52734aeb4c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250903 08:07:49.024034 14282 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 5b996e3f77d84707b4bfc52734aeb4c9 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 08:07:49.024240 14282 raft_consensus.cc:695] T 00000000000000000000000000000000 P 5b996e3f77d84707b4bfc52734aeb4c9 [term 1 LEADER]: Becoming Leader. State: Replica: 5b996e3f77d84707b4bfc52734aeb4c9, State: Running, Role: LEADER
I20250903 08:07:49.024523 14282 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b996e3f77d84707b4bfc52734aeb4c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b996e3f77d84707b4bfc52734aeb4c9" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 46669 } }
I20250903 08:07:49.025024 14277 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b996e3f77d84707b4bfc52734aeb4c9 [sys.catalog]: configured and running, proceeding with master startup.
I20250903 08:07:49.025978 14283 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b996e3f77d84707b4bfc52734aeb4c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b996e3f77d84707b4bfc52734aeb4c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b996e3f77d84707b4bfc52734aeb4c9" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 46669 } } }
I20250903 08:07:49.026104 14283 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b996e3f77d84707b4bfc52734aeb4c9 [sys.catalog]: This master's current role is: LEADER
I20250903 08:07:49.026343 14284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b996e3f77d84707b4bfc52734aeb4c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b996e3f77d84707b4bfc52734aeb4c9. Latest consensus state: current_term: 1 leader_uuid: "5b996e3f77d84707b4bfc52734aeb4c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b996e3f77d84707b4bfc52734aeb4c9" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 46669 } } }
I20250903 08:07:49.026425 14291 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250903 08:07:49.026446 14284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b996e3f77d84707b4bfc52734aeb4c9 [sys.catalog]: This master's current role is: LEADER
I20250903 08:07:49.029726 14291 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250903 08:07:49.035063 14291 catalog_manager.cc:1349] Generated new cluster ID: 8b601a5dc73842d680980308946fe36e
I20250903 08:07:49.035142 14291 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250903 08:07:49.052800 14291 catalog_manager.cc:1372] Generated new certificate authority record
I20250903 08:07:49.053766 14291 catalog_manager.cc:1506] Loading token signing keys...
I20250903 08:07:49.061060 14291 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 5b996e3f77d84707b4bfc52734aeb4c9: Generated new TSK 0
I20250903 08:07:49.061735 14291 catalog_manager.cc:1516] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.201.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.201.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.201.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.201.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.201.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.201.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.201.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.201.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/krb5kdc/kudu.keytab.
I20250903 08:07:49.110961 13095 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUML90r/build/debug/bin/kudu
/tmp/dist-test-taskUML90r/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.201.193:0
--local_ip_for_outbound_sockets=127.12.201.193
--webserver_interface=127.12.201.193
--webserver_port=0
--tserver_master_addrs=127.12.201.254:46669
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.201.212:37769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 08:07:49.215718 14305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:07:49.215916 14305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:07:49.215984 14305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:07:49.219163 14305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 08:07:49.219225 14305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:07:49.219290 14305 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20250903 08:07:49.219367 14305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.201.193
I20250903 08:07:49.224400 14305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.201.212:37769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.201.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.201.193
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.201.254:46669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.201.193
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 07:50:08 UTC on 5fd53c4cbb9d
build id 7874
I20250903 08:07:49.225471 14305 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:07:49.226351 14305 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:07:49.232509 14313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:07:49.232542 14311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:07:49.232487 14310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:07:49.232741 14305 server_base.cc:1047] running on GCE node
I20250903 08:07:49.233075 14305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:07:49.233580 14305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:07:49.234737 14305 hybrid_clock.cc:648] HybridClock initialized: now 1756886869234722 us; error 27 us; skew 500 ppm
Sep 03 08:07:49 dist-test-slave-27r9 krb5kdc[14183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886869, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 08:07:49.237390 14305 init.cc:377] Logged in from keytab as kudu/127.12.201.193@KRBTEST.COM (short username kudu)
I20250903 08:07:49.238390 14305 webserver.cc:480] Webserver started at http://127.12.201.193:33319/ using document root <none> and password file <none>
I20250903 08:07:49.238919 14305 fs_manager.cc:362] Metadata directory not provided
I20250903 08:07:49.238991 14305 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:07:49.239180 14305 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:07:49.240855 14305 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/minicluster-data/ts-0/data/instance:
uuid: "dc9b942bcd6349a6adb9faaff24b94b3"
format_stamp: "Formatted at 2025-09-03 08:07:49 on dist-test-slave-27r9"
server_key: "f6a3a896b0c454d252b880bf0010076d"
server_key_iv: "8cf67c236f7c0feff9e620e92d2bcf31"
server_key_version: "encryptionkey@0"
I20250903 08:07:49.241335 14305 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/minicluster-data/ts-0/wal/instance:
uuid: "dc9b942bcd6349a6adb9faaff24b94b3"
format_stamp: "Formatted at 2025-09-03 08:07:49 on dist-test-slave-27r9"
server_key: "f6a3a896b0c454d252b880bf0010076d"
server_key_iv: "8cf67c236f7c0feff9e620e92d2bcf31"
server_key_version: "encryptionkey@0"
I20250903 08:07:49.244632 14305 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250903 08:07:49.246752 14320 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:07:49.247781 14305 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250903 08:07:49.247875 14305 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/minicluster-data/ts-0/wal
uuid: "dc9b942bcd6349a6adb9faaff24b94b3"
format_stamp: "Formatted at 2025-09-03 08:07:49 on dist-test-slave-27r9"
server_key: "f6a3a896b0c454d252b880bf0010076d"
server_key_iv: "8cf67c236f7c0feff9e620e92d2bcf31"
server_key_version: "encryptionkey@0"
I20250903 08:07:49.247965 14305 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:07:49.266045 14305 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:07:49.269279 14305 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:07:49.269487 14305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:07:49.271005 14305 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 08:07:49.271061 14305 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 08:07:49.271122 14305 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 08:07:49.271157 14305 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 03 08:07:49 dist-test-slave-27r9 krb5kdc[14183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886869, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.193@KRBTEST.COM for kudu/127.12.201.254@KRBTEST.COM
I20250903 08:07:49.282526 14435 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.201.193:41925 every 8 connection(s)
I20250903 08:07:49.282501 14305 rpc_server.cc:307] RPC server started. Bound to: 127.12.201.193:41925
I20250903 08:07:49.283241 14280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:49.271803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.193:33733 (local address 127.12.201.254:46669)
0903 08:07:49.271929 (+ 126us) server_negotiation.cc:207] Beginning negotiation
0903 08:07:49.271933 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:07:49.272801 (+ 868us) server_negotiation.cc:408] Connection header received
0903 08:07:49.273684 (+ 883us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:07:49.273688 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:07:49.273752 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:07:49.273828 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:07:49.275422 (+ 1594us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:49.276056 (+ 634us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:07:49.276743 (+ 687us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:49.276927 (+ 184us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:49.279548 (+ 2621us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:07:49.279579 (+ 31us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:07:49.279583 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:07:49.279613 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:07:49.281257 (+ 1644us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:07:49.281771 (+ 514us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:07:49.281774 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:07:49.281775 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:07:49.281818 (+ 43us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:07:49.282078 (+ 260us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:07:49.282082 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:07:49.282084 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:07:49.282260 (+ 176us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:07:49.282359 (+ 99us) server_negotiation.cc:1036] Waiting for connection context
0903 08:07:49.282916 (+ 557us) server_negotiation.cc:300] Negotiation successful
0903 08:07:49.283075 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20250903 08:07:49.283674 14305 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/minicluster-data/ts-0/data/info.pb
I20250903 08:07:49.283828 14329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:49.272114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:46669 (local address 127.12.201.193:33733)
0903 08:07:49.272635 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0903 08:07:49.272675 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0903 08:07:49.273465 (+ 790us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:07:49.274016 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:07:49.274028 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:07:49.274592 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:07:49.275270 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:49.275286 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:49.276177 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:07:49.276181 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:07:49.276631 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:49.276639 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:49.276829 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:49.277517 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:07:49.277557 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:07:49.279392 (+ 1835us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:07:49.281365 (+ 1973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:07:49.281370 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:07:49.281382 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:07:49.281666 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:07:49.281898 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:07:49.281901 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:07:49.281903 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:07:49.281995 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:07:49.282376 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:07:49.282383 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:07:49.282648 (+ 265us) client_negotiation.cc:769] Sending connection context
0903 08:07:49.282867 (+ 219us) client_negotiation.cc:240] Negotiation successful
0903 08:07:49.283158 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":142,"threads_started":1}
I20250903 08:07:49.285358 14436 heartbeater.cc:344] Connected to a master server at 127.12.201.254:46669
I20250903 08:07:49.285507 14436 heartbeater.cc:461] Registering TS with master...
I20250903 08:07:49.286075 14436 heartbeater.cc:507] Master 127.12.201.254:46669 requested a full tablet report, sending...
I20250903 08:07:49.286480 13095 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUML90r/build/debug/bin/kudu as pid 14305
I20250903 08:07:49.286578 13095 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/minicluster-data/ts-0/wal/instance
I20250903 08:07:49.286808 13095 external_mini_cluster.cc:1468] Setting key dc8982bc9aee7ef87892aa952a3a2d47
I20250903 08:07:49.288092 14231 ts_manager.cc:194] Registered new tserver with Master: dc9b942bcd6349a6adb9faaff24b94b3 (127.12.201.193:41925)
I20250903 08:07:49.289680 14231 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.201.193@KRBTEST.COM'} at 127.12.201.193:33733
I20250903 08:07:49.295090 14280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:49.287582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.193:46331 (local address 127.12.201.254:46669)
0903 08:07:49.287698 (+ 116us) server_negotiation.cc:207] Beginning negotiation
0903 08:07:49.287702 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:07:49.287915 (+ 213us) server_negotiation.cc:408] Connection header received
0903 08:07:49.288061 (+ 146us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:07:49.288064 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:07:49.288102 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:07:49.288171 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:07:49.288851 (+ 680us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:49.290592 (+ 1741us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:07:49.291312 (+ 720us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:49.291558 (+ 246us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:49.292331 (+ 773us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:07:49.292350 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:07:49.292352 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:07:49.292373 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:07:49.293738 (+ 1365us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:07:49.294104 (+ 366us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:07:49.294110 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:07:49.294112 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:07:49.294151 (+ 39us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:07:49.294367 (+ 216us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:07:49.294370 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:07:49.294372 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:07:49.294506 (+ 134us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:07:49.294593 (+ 87us) server_negotiation.cc:1036] Waiting for connection context
0903 08:07:49.294823 (+ 230us) server_negotiation.cc:300] Negotiation successful
0903 08:07:49.294927 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20250903 08:07:49.295111 14442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:49.287596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:46669 (local address 127.12.201.193:46331)
0903 08:07:49.287844 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0903 08:07:49.287858 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0903 08:07:49.287970 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:07:49.288316 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:07:49.288319 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:07:49.288520 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:07:49.288721 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:49.288727 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:49.290712 (+ 1985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:07:49.290719 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:07:49.291194 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:49.291203 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:49.291320 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:49.291930 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:07:49.291942 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:07:49.292209 (+ 267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:07:49.293846 (+ 1637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:07:49.293849 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:07:49.293851 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:07:49.294015 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:07:49.294240 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:07:49.294243 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:07:49.294244 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:07:49.294289 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:07:49.294593 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:07:49.294597 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:07:49.294676 (+ 79us) client_negotiation.cc:769] Sending connection context
0903 08:07:49.294765 (+ 89us) client_negotiation.cc:240] Negotiation successful
0903 08:07:49.294899 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":97,"threads_started":1}
WARNING: no policy specified for kudu/127.12.201.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.201.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.201.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.201.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.201.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.201.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.201.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.201.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/krb5kdc/kudu.keytab.
I20250903 08:07:49.332202 13095 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUML90r/build/debug/bin/kudu
/tmp/dist-test-taskUML90r/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.201.194:0
--local_ip_for_outbound_sockets=127.12.201.194
--webserver_interface=127.12.201.194
--webserver_port=0
--tserver_master_addrs=127.12.201.254:46669
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.201.212:37769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 08:07:49.436535 14447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:07:49.436736 14447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:07:49.436795 14447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:07:49.439952 14447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 08:07:49.440008 14447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:07:49.440058 14447 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20250903 08:07:49.440104 14447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.201.194
I20250903 08:07:49.444660 14447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.201.212:37769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.201.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.201.194
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.201.254:46669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14447
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.201.194
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 07:50:08 UTC on 5fd53c4cbb9d
build id 7874
I20250903 08:07:49.445606 14447 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:07:49.446367 14447 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:07:49.452527 14453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:07:49.452534 14452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:07:49.452556 14455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:07:49.452766 14447 server_base.cc:1047] running on GCE node
I20250903 08:07:49.453130 14447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:07:49.453645 14447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:07:49.454792 14447 hybrid_clock.cc:648] HybridClock initialized: now 1756886869454772 us; error 39 us; skew 500 ppm
Sep 03 08:07:49 dist-test-slave-27r9 krb5kdc[14183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886869, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 08:07:49.457399 14447 init.cc:377] Logged in from keytab as kudu/127.12.201.194@KRBTEST.COM (short username kudu)
I20250903 08:07:49.458397 14447 webserver.cc:480] Webserver started at http://127.12.201.194:34067/ using document root <none> and password file <none>
I20250903 08:07:49.458911 14447 fs_manager.cc:362] Metadata directory not provided
I20250903 08:07:49.458964 14447 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:07:49.459164 14447 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:07:49.460812 14447 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/minicluster-data/ts-1/data/instance:
uuid: "b9be75cb719d4a1181d0d27104544cdc"
format_stamp: "Formatted at 2025-09-03 08:07:49 on dist-test-slave-27r9"
server_key: "21736cf5a878b335c270de0860604ad6"
server_key_iv: "c10c27897e2621b533b7c60a3915cfbc"
server_key_version: "encryptionkey@0"
I20250903 08:07:49.461288 14447 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/minicluster-data/ts-1/wal/instance:
uuid: "b9be75cb719d4a1181d0d27104544cdc"
format_stamp: "Formatted at 2025-09-03 08:07:49 on dist-test-slave-27r9"
server_key: "21736cf5a878b335c270de0860604ad6"
server_key_iv: "c10c27897e2621b533b7c60a3915cfbc"
server_key_version: "encryptionkey@0"
I20250903 08:07:49.464495 14447 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20250903 08:07:49.466485 14462 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:07:49.467478 14447 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250903 08:07:49.467576 14447 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/minicluster-data/ts-1/wal
uuid: "b9be75cb719d4a1181d0d27104544cdc"
format_stamp: "Formatted at 2025-09-03 08:07:49 on dist-test-slave-27r9"
server_key: "21736cf5a878b335c270de0860604ad6"
server_key_iv: "c10c27897e2621b533b7c60a3915cfbc"
server_key_version: "encryptionkey@0"
I20250903 08:07:49.467657 14447 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:07:49.492975 14447 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:07:49.495985 14447 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:07:49.496165 14447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:07:49.497666 14447 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 08:07:49.497720 14447 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 08:07:49.497779 14447 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 08:07:49.497815 14447 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 03 08:07:49 dist-test-slave-27r9 krb5kdc[14183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886869, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.194@KRBTEST.COM for kudu/127.12.201.254@KRBTEST.COM
I20250903 08:07:49.508286 14447 rpc_server.cc:307] RPC server started. Bound to: 127.12.201.194:35997
I20250903 08:07:49.508320 14577 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.201.194:35997 every 8 connection(s)
I20250903 08:07:49.509415 14447 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/minicluster-data/ts-1/data/info.pb
I20250903 08:07:49.510627 14280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:49.498259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.194:53125 (local address 127.12.201.254:46669)
0903 08:07:49.498379 (+ 120us) server_negotiation.cc:207] Beginning negotiation
0903 08:07:49.498383 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:07:49.499277 (+ 894us) server_negotiation.cc:408] Connection header received
0903 08:07:49.500133 (+ 856us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:07:49.500139 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:07:49.500204 (+ 65us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:07:49.500313 (+ 109us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:07:49.501813 (+ 1500us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:49.502663 (+ 850us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:07:49.503545 (+ 882us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:49.503714 (+ 169us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:49.506685 (+ 2971us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:07:49.506711 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:07:49.506717 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:07:49.506753 (+ 36us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:07:49.508715 (+ 1962us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:07:49.509165 (+ 450us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:07:49.509169 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:07:49.509171 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:07:49.509223 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:07:49.509487 (+ 264us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:07:49.509490 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:07:49.509492 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:07:49.509682 (+ 190us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:07:49.509807 (+ 125us) server_negotiation.cc:1036] Waiting for connection context
0903 08:07:49.510330 (+ 523us) server_negotiation.cc:300] Negotiation successful
0903 08:07:49.510451 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20250903 08:07:49.511251 14471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:49.498601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:46669 (local address 127.12.201.194:53125)
0903 08:07:49.499119 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0903 08:07:49.499182 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0903 08:07:49.499909 (+ 727us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:07:49.500458 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:07:49.500468 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:07:49.500963 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:07:49.501666 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:49.501679 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:49.502789 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:07:49.502794 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:07:49.503438 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:49.503448 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:49.503638 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:49.504308 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:07:49.504340 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:07:49.506556 (+ 2216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:07:49.508818 (+ 2262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:07:49.508822 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:07:49.508830 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:07:49.509055 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:07:49.509310 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:07:49.509312 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:07:49.509314 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:07:49.509389 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:07:49.509807 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:07:49.509813 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:07:49.510058 (+ 245us) client_negotiation.cc:769] Sending connection context
0903 08:07:49.510281 (+ 223us) client_negotiation.cc:240] Negotiation successful
0903 08:07:49.510546 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":146,"threads_started":1}
I20250903 08:07:49.512316 14578 heartbeater.cc:344] Connected to a master server at 127.12.201.254:46669
I20250903 08:07:49.512478 14578 heartbeater.cc:461] Registering TS with master...
I20250903 08:07:49.512972 14578 heartbeater.cc:507] Master 127.12.201.254:46669 requested a full tablet report, sending...
I20250903 08:07:49.513927 14230 ts_manager.cc:194] Registered new tserver with Master: b9be75cb719d4a1181d0d27104544cdc (127.12.201.194:35997)
I20250903 08:07:49.514684 14230 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.201.194@KRBTEST.COM'} at 127.12.201.194:53125
I20250903 08:07:49.517671 13095 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUML90r/build/debug/bin/kudu as pid 14447
I20250903 08:07:49.517792 13095 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/minicluster-data/ts-1/wal/instance
I20250903 08:07:49.518042 13095 external_mini_cluster.cc:1468] Setting key 0b5946df8252991fe85af4224a4a60fc
I20250903 08:07:49.520536 14280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:49.514339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.194:59873 (local address 127.12.201.254:46669)
0903 08:07:49.514455 (+ 116us) server_negotiation.cc:207] Beginning negotiation
0903 08:07:49.514458 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 08:07:49.514629 (+ 171us) server_negotiation.cc:408] Connection header received
0903 08:07:49.514737 (+ 108us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:07:49.514739 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:07:49.514783 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:07:49.514871 (+ 88us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:07:49.515479 (+ 608us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:49.515935 (+ 456us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:07:49.516500 (+ 565us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:49.516695 (+ 195us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:49.517446 (+ 751us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:07:49.517460 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:07:49.517462 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:07:49.517482 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:07:49.519092 (+ 1610us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:07:49.519558 (+ 466us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:07:49.519562 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:07:49.519563 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:07:49.519606 (+ 43us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:07:49.519851 (+ 245us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:07:49.519853 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:07:49.519854 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:07:49.519994 (+ 140us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:07:49.520069 (+ 75us) server_negotiation.cc:1036] Waiting for connection context
0903 08:07:49.520338 (+ 269us) server_negotiation.cc:300] Negotiation successful
0903 08:07:49.520429 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20250903 08:07:49.520711 14584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:49.514275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:46669 (local address 127.12.201.194:59873)
0903 08:07:49.514554 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0903 08:07:49.514568 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0903 08:07:49.514662 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:07:49.514892 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:07:49.514895 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:07:49.515107 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:07:49.515367 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:49.515374 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:49.516041 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:07:49.516044 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:07:49.516400 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:49.516407 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:49.516508 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:49.517021 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:07:49.517033 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:07:49.517309 (+ 276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:07:49.519220 (+ 1911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:07:49.519223 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:07:49.519225 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:07:49.519444 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:07:49.519693 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:07:49.519698 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:07:49.519704 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:07:49.519772 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:07:49.520082 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:07:49.520088 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:07:49.520181 (+ 93us) client_negotiation.cc:769] Sending connection context
0903 08:07:49.520293 (+ 112us) client_negotiation.cc:240] Negotiation successful
0903 08:07:49.520444 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":81,"threads_started":1}
WARNING: no policy specified for kudu/127.12.201.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.201.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.201.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.201.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.201.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.201.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.201.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.201.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/krb5kdc/kudu.keytab.
I20250903 08:07:49.562057 13095 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUML90r/build/debug/bin/kudu
/tmp/dist-test-taskUML90r/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.201.195:0
--local_ip_for_outbound_sockets=127.12.201.195
--webserver_interface=127.12.201.195
--webserver_port=0
--tserver_master_addrs=127.12.201.254:46669
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.201.212:37769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 08:07:49.666960 14589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:07:49.667186 14589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:07:49.667246 14589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:07:49.670436 14589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 08:07:49.670495 14589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:07:49.670547 14589 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20250903 08:07:49.670590 14589 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.201.195
I20250903 08:07:49.674722 14589 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.201.212:37769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.201.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.201.195
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.201.254:46669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14589
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.201.195
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 07:50:08 UTC on 5fd53c4cbb9d
build id 7874
I20250903 08:07:49.675721 14589 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:07:49.676520 14589 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:07:49.682783 14597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:07:49.682749 14595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:07:49.682853 14589 server_base.cc:1047] running on GCE node
W20250903 08:07:49.682757 14594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:07:49.683367 14589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:07:49.683869 14589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:07:49.685022 14589 hybrid_clock.cc:648] HybridClock initialized: now 1756886869684999 us; error 37 us; skew 500 ppm
Sep 03 08:07:49 dist-test-slave-27r9 krb5kdc[14183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886869, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 08:07:49.687709 14589 init.cc:377] Logged in from keytab as kudu/127.12.201.195@KRBTEST.COM (short username kudu)
I20250903 08:07:49.688666 14589 webserver.cc:480] Webserver started at http://127.12.201.195:33555/ using document root <none> and password file <none>
I20250903 08:07:49.689158 14589 fs_manager.cc:362] Metadata directory not provided
I20250903 08:07:49.689211 14589 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:07:49.689360 14589 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:07:49.690970 14589 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/minicluster-data/ts-2/data/instance:
uuid: "34e13456e0c24db6a9b9b04a81897d08"
format_stamp: "Formatted at 2025-09-03 08:07:49 on dist-test-slave-27r9"
server_key: "c90fd8821e12ee49e990f04cb5438c50"
server_key_iv: "5aec6902ffd4d6dd01c76d0197bfb2e6"
server_key_version: "encryptionkey@0"
I20250903 08:07:49.691504 14589 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/minicluster-data/ts-2/wal/instance:
uuid: "34e13456e0c24db6a9b9b04a81897d08"
format_stamp: "Formatted at 2025-09-03 08:07:49 on dist-test-slave-27r9"
server_key: "c90fd8821e12ee49e990f04cb5438c50"
server_key_iv: "5aec6902ffd4d6dd01c76d0197bfb2e6"
server_key_version: "encryptionkey@0"
I20250903 08:07:49.694669 14589 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20250903 08:07:49.696725 14604 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:07:49.697696 14589 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20250903 08:07:49.697785 14589 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/minicluster-data/ts-2/wal
uuid: "34e13456e0c24db6a9b9b04a81897d08"
format_stamp: "Formatted at 2025-09-03 08:07:49 on dist-test-slave-27r9"
server_key: "c90fd8821e12ee49e990f04cb5438c50"
server_key_iv: "5aec6902ffd4d6dd01c76d0197bfb2e6"
server_key_version: "encryptionkey@0"
I20250903 08:07:49.697856 14589 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:07:49.709154 14589 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:07:49.711588 14589 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:07:49.711728 14589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:07:49.712966 14589 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 08:07:49.713016 14589 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 08:07:49.713073 14589 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 08:07:49.713104 14589 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 03 08:07:49 dist-test-slave-27r9 krb5kdc[14183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886869, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.195@KRBTEST.COM for kudu/127.12.201.254@KRBTEST.COM
I20250903 08:07:49.723269 14589 rpc_server.cc:307] RPC server started. Bound to: 127.12.201.195:35365
I20250903 08:07:49.723308 14719 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.201.195:35365 every 8 connection(s)
I20250903 08:07:49.724335 14589 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/minicluster-data/ts-2/data/info.pb
I20250903 08:07:49.725473 14280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:49.713648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.195:43119 (local address 127.12.201.254:46669)
0903 08:07:49.713779 (+ 131us) server_negotiation.cc:207] Beginning negotiation
0903 08:07:49.713784 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 08:07:49.714457 (+ 673us) server_negotiation.cc:408] Connection header received
0903 08:07:49.715315 (+ 858us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:07:49.715322 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:07:49.715388 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:07:49.715503 (+ 115us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:07:49.717015 (+ 1512us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:49.717890 (+ 875us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:07:49.718834 (+ 944us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:49.719111 (+ 277us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:49.721717 (+ 2606us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:07:49.721741 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:07:49.721747 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:07:49.721778 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:07:49.723504 (+ 1726us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:07:49.724097 (+ 593us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:07:49.724100 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:07:49.724101 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:07:49.724149 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:07:49.724507 (+ 358us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:07:49.724509 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:07:49.724511 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:07:49.724671 (+ 160us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:07:49.724777 (+ 106us) server_negotiation.cc:1036] Waiting for connection context
0903 08:07:49.725250 (+ 473us) server_negotiation.cc:300] Negotiation successful
0903 08:07:49.725348 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20250903 08:07:49.726006 14613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:49.713884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:46669 (local address 127.12.201.195:43119)
0903 08:07:49.714297 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0903 08:07:49.714338 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0903 08:07:49.715123 (+ 785us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:07:49.715657 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:07:49.715665 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:07:49.716141 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:07:49.716825 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:49.716844 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:49.718020 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:07:49.718025 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:07:49.718686 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:49.718697 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:49.718889 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:49.719527 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:07:49.719557 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:07:49.721552 (+ 1995us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:07:49.723620 (+ 2068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:07:49.723625 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:07:49.723634 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:07:49.723916 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:07:49.724244 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:07:49.724250 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:07:49.724255 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:07:49.724373 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:07:49.724775 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:07:49.724781 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:07:49.725012 (+ 231us) client_negotiation.cc:769] Sending connection context
0903 08:07:49.725182 (+ 170us) client_negotiation.cc:240] Negotiation successful
0903 08:07:49.725423 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":106,"threads_started":1}
I20250903 08:07:49.727089 14720 heartbeater.cc:344] Connected to a master server at 127.12.201.254:46669
I20250903 08:07:49.727329 14720 heartbeater.cc:461] Registering TS with master...
I20250903 08:07:49.727849 13095 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUML90r/build/debug/bin/kudu as pid 14589
I20250903 08:07:49.727907 14720 heartbeater.cc:507] Master 127.12.201.254:46669 requested a full tablet report, sending...
I20250903 08:07:49.727967 13095 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/minicluster-data/ts-2/wal/instance
I20250903 08:07:49.728257 13095 external_mini_cluster.cc:1468] Setting key e325f2a83438c463c3bada669f69a67a
I20250903 08:07:49.728991 14230 ts_manager.cc:194] Registered new tserver with Master: 34e13456e0c24db6a9b9b04a81897d08 (127.12.201.195:35365)
I20250903 08:07:49.729732 14230 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.201.195@KRBTEST.COM'} at 127.12.201.195:43119
I20250903 08:07:49.730201 13095 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250903 08:07:49.736284 14726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:49.729439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:46669 (local address 127.12.201.195:45977)
0903 08:07:49.729709 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0903 08:07:49.729724 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0903 08:07:49.729823 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:07:49.730138 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:07:49.730143 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:07:49.730333 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:07:49.730602 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:49.730611 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:49.731611 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:07:49.731614 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:07:49.732008 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:49.732015 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:49.732125 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:49.732734 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:07:49.732754 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:07:49.733148 (+ 394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:07:49.734920 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:07:49.734923 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:07:49.734925 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:07:49.735118 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:07:49.735382 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:07:49.735384 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:07:49.735386 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:07:49.735430 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:07:49.735798 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:07:49.735800 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:07:49.735871 (+ 71us) client_negotiation.cc:769] Sending connection context
0903 08:07:49.735961 (+ 90us) client_negotiation.cc:240] Negotiation successful
0903 08:07:49.736089 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":86,"threads_started":1}
I20250903 08:07:49.736284 14280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:49.729466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.195:45977 (local address 127.12.201.254:46669)
0903 08:07:49.729582 (+ 116us) server_negotiation.cc:207] Beginning negotiation
0903 08:07:49.729586 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:07:49.729799 (+ 213us) server_negotiation.cc:408] Connection header received
0903 08:07:49.729899 (+ 100us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:07:49.729903 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:07:49.729956 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:07:49.730027 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:07:49.730734 (+ 707us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:49.731488 (+ 754us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:07:49.732118 (+ 630us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:49.732360 (+ 242us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:49.733264 (+ 904us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:07:49.733279 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:07:49.733281 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:07:49.733303 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:07:49.734806 (+ 1503us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:07:49.735215 (+ 409us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:07:49.735222 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:07:49.735227 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:07:49.735283 (+ 56us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:07:49.735515 (+ 232us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:07:49.735521 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:07:49.735526 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:07:49.735714 (+ 188us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:07:49.735818 (+ 104us) server_negotiation.cc:1036] Waiting for connection context
0903 08:07:49.736038 (+ 220us) server_negotiation.cc:300] Negotiation successful
0903 08:07:49.736141 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
Sep 03 08:07:49 dist-test-slave-27r9 krb5kdc[14183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886869, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 03 08:07:49 dist-test-slave-27r9 krb5kdc[14183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886869, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.201.254@KRBTEST.COM
I20250903 08:07:49.753948 14280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:49.747029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52336 (local address 127.12.201.254:46669)
0903 08:07:49.747140 (+ 111us) server_negotiation.cc:207] Beginning negotiation
0903 08:07:49.747144 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:07:49.747155 (+ 11us) server_negotiation.cc:408] Connection header received
0903 08:07:49.747194 (+ 39us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:07:49.747196 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:07:49.747234 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:07:49.747304 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:07:49.748027 (+ 723us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:49.748532 (+ 505us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:07:49.749081 (+ 549us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:49.749201 (+ 120us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:49.751224 (+ 2023us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:07:49.751241 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:07:49.751243 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:07:49.751265 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:07:49.752656 (+ 1391us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:07:49.753020 (+ 364us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:07:49.753023 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:07:49.753024 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:07:49.753066 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:07:49.753222 (+ 156us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:07:49.753224 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:07:49.753226 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:07:49.753356 (+ 130us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:07:49.753465 (+ 109us) server_negotiation.cc:1036] Waiting for connection context
0903 08:07:49.753645 (+ 180us) server_negotiation.cc:300] Negotiation successful
0903 08:07:49.753764 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20250903 08:07:49.756671 14230 catalog_manager.cc:2232] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52336:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 08:07:49.758970 14230 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 08:07:49.771158 14740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:49.766815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.194:35997 (local address 127.0.0.1:34316)
0903 08:07:49.767829 (+ 1014us) negotiation.cc:107] Waiting for socket to connect
0903 08:07:49.767843 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0903 08:07:49.767914 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:07:49.768667 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:07:49.768671 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:07:49.768684 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 08:07:49.768896 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:49.768902 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:49.769837 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:07:49.769841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:07:49.770640 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:49.770650 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:49.770755 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:49.770807 (+ 52us) client_negotiation.cc:769] Sending connection context
0903 08:07:49.770888 (+ 81us) client_negotiation.cc:240] Negotiation successful
0903 08:07:49.770967 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":927,"thread_start_us":80,"threads_started":1}
I20250903 08:07:49.772063 14742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:49.767473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34316 (local address 127.12.201.194:35997)
0903 08:07:49.768251 (+ 778us) server_negotiation.cc:207] Beginning negotiation
0903 08:07:49.768256 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 08:07:49.768281 (+ 25us) server_negotiation.cc:408] Connection header received
0903 08:07:49.768361 (+ 80us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:07:49.768366 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:07:49.768535 (+ 169us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:07:49.768669 (+ 134us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 08:07:49.769004 (+ 335us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:49.769730 (+ 726us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:07:49.771076 (+ 1346us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:49.771650 (+ 574us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:49.771787 (+ 137us) server_negotiation.cc:1036] Waiting for connection context
0903 08:07:49.771849 (+ 62us) server_negotiation.cc:300] Negotiation successful
0903 08:07:49.771928 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":668,"thread_start_us":110,"threads_started":1}
I20250903 08:07:49.772310 14739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:49.766866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45306 (local address 127.12.201.195:35365)
0903 08:07:49.767685 (+ 819us) server_negotiation.cc:207] Beginning negotiation
0903 08:07:49.767692 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0903 08:07:49.767720 (+ 28us) server_negotiation.cc:408] Connection header received
0903 08:07:49.767957 (+ 237us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:07:49.767964 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:07:49.768173 (+ 209us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:07:49.768326 (+ 153us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 08:07:49.768811 (+ 485us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:49.770030 (+ 1219us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:07:49.771449 (+ 1419us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:49.771938 (+ 489us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:49.772031 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
0903 08:07:49.772096 (+ 65us) server_negotiation.cc:300] Negotiation successful
0903 08:07:49.772183 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":721,"thread_start_us":146,"threads_started":1}
I20250903 08:07:49.772511 14737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:49.767100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.193:41925 (local address 127.0.0.1:49938)
0903 08:07:49.767860 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0903 08:07:49.767875 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0903 08:07:49.767960 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:07:49.768505 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:07:49.768508 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:07:49.768524 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 08:07:49.768759 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:49.768766 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:49.770160 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:07:49.770163 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:07:49.771056 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:49.771065 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:49.771157 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:49.771177 (+ 20us) client_negotiation.cc:769] Sending connection context
0903 08:07:49.771230 (+ 53us) client_negotiation.cc:240] Negotiation successful
0903 08:07:49.771299 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":697,"thread_start_us":104,"threads_started":1}
I20250903 08:07:49.772541 14738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:49.766727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.195:35365 (local address 127.0.0.1:45306)
0903 08:07:49.767564 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0903 08:07:49.767606 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0903 08:07:49.767755 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:07:49.768341 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:07:49.768346 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:07:49.768371 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 08:07:49.768676 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:49.768684 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:49.770187 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:07:49.770190 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:07:49.771065 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:49.771075 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:49.771176 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:49.771193 (+ 17us) client_negotiation.cc:769] Sending connection context
0903 08:07:49.771241 (+ 48us) client_negotiation.cc:240] Negotiation successful
0903 08:07:49.771300 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":671,"thread_start_us":86,"threads_started":1}
I20250903 08:07:49.772746 14741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:49.767343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49938 (local address 127.12.201.193:41925)
0903 08:07:49.767685 (+ 342us) server_negotiation.cc:207] Beginning negotiation
0903 08:07:49.767691 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0903 08:07:49.767998 (+ 307us) server_negotiation.cc:408] Connection header received
0903 08:07:49.768075 (+ 77us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:07:49.768079 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:07:49.768228 (+ 149us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:07:49.768355 (+ 127us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 08:07:49.768893 (+ 538us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:49.770045 (+ 1152us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:07:49.771499 (+ 1454us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:49.772000 (+ 501us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:49.772079 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
0903 08:07:49.772143 (+ 64us) server_negotiation.cc:300] Negotiation successful
0903 08:07:49.772242 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":89,"threads_started":1}
I20250903 08:07:49.774542 14512 tablet_service.cc:1468] Processing CreateTablet for tablet 410b298ec531468680fbaa1fa3e1f7de (DEFAULT_TABLE table=test-table [id=e9907ae074a24f61a5c1417dc679ce7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:07:49.774767 14370 tablet_service.cc:1468] Processing CreateTablet for tablet 410b298ec531468680fbaa1fa3e1f7de (DEFAULT_TABLE table=test-table [id=e9907ae074a24f61a5c1417dc679ce7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:07:49.774765 14654 tablet_service.cc:1468] Processing CreateTablet for tablet 410b298ec531468680fbaa1fa3e1f7de (DEFAULT_TABLE table=test-table [id=e9907ae074a24f61a5c1417dc679ce7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:07:49.775502 14512 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 410b298ec531468680fbaa1fa3e1f7de. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:07:49.775647 14654 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 410b298ec531468680fbaa1fa3e1f7de. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:07:49.775647 14370 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 410b298ec531468680fbaa1fa3e1f7de. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:07:49.780826 14743 tablet_bootstrap.cc:492] T 410b298ec531468680fbaa1fa3e1f7de P b9be75cb719d4a1181d0d27104544cdc: Bootstrap starting.
I20250903 08:07:49.782037 14744 tablet_bootstrap.cc:492] T 410b298ec531468680fbaa1fa3e1f7de P 34e13456e0c24db6a9b9b04a81897d08: Bootstrap starting.
I20250903 08:07:49.783273 14745 tablet_bootstrap.cc:492] T 410b298ec531468680fbaa1fa3e1f7de P dc9b942bcd6349a6adb9faaff24b94b3: Bootstrap starting.
I20250903 08:07:49.783507 14743 tablet_bootstrap.cc:654] T 410b298ec531468680fbaa1fa3e1f7de P b9be75cb719d4a1181d0d27104544cdc: Neither blocks nor log segments found. Creating new log.
I20250903 08:07:49.784071 14744 tablet_bootstrap.cc:654] T 410b298ec531468680fbaa1fa3e1f7de P 34e13456e0c24db6a9b9b04a81897d08: Neither blocks nor log segments found. Creating new log.
I20250903 08:07:49.785214 14745 tablet_bootstrap.cc:654] T 410b298ec531468680fbaa1fa3e1f7de P dc9b942bcd6349a6adb9faaff24b94b3: Neither blocks nor log segments found. Creating new log.
I20250903 08:07:49.785327 14744 log.cc:826] T 410b298ec531468680fbaa1fa3e1f7de P 34e13456e0c24db6a9b9b04a81897d08: Log is configured to *not* fsync() on all Append() calls
I20250903 08:07:49.785327 14743 log.cc:826] T 410b298ec531468680fbaa1fa3e1f7de P b9be75cb719d4a1181d0d27104544cdc: Log is configured to *not* fsync() on all Append() calls
I20250903 08:07:49.785916 14745 log.cc:826] T 410b298ec531468680fbaa1fa3e1f7de P dc9b942bcd6349a6adb9faaff24b94b3: Log is configured to *not* fsync() on all Append() calls
I20250903 08:07:49.787235 14744 tablet_bootstrap.cc:492] T 410b298ec531468680fbaa1fa3e1f7de P 34e13456e0c24db6a9b9b04a81897d08: No bootstrap required, opened a new log
I20250903 08:07:49.787235 14743 tablet_bootstrap.cc:492] T 410b298ec531468680fbaa1fa3e1f7de P b9be75cb719d4a1181d0d27104544cdc: No bootstrap required, opened a new log
I20250903 08:07:49.787395 14745 tablet_bootstrap.cc:492] T 410b298ec531468680fbaa1fa3e1f7de P dc9b942bcd6349a6adb9faaff24b94b3: No bootstrap required, opened a new log
I20250903 08:07:49.787431 14744 ts_tablet_manager.cc:1397] T 410b298ec531468680fbaa1fa3e1f7de P 34e13456e0c24db6a9b9b04a81897d08: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20250903 08:07:49.787431 14743 ts_tablet_manager.cc:1397] T 410b298ec531468680fbaa1fa3e1f7de P b9be75cb719d4a1181d0d27104544cdc: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20250903 08:07:49.787564 14745 ts_tablet_manager.cc:1397] T 410b298ec531468680fbaa1fa3e1f7de P dc9b942bcd6349a6adb9faaff24b94b3: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250903 08:07:49.790396 14744 raft_consensus.cc:357] T 410b298ec531468680fbaa1fa3e1f7de P 34e13456e0c24db6a9b9b04a81897d08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34e13456e0c24db6a9b9b04a81897d08" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 35365 } } peers { permanent_uuid: "b9be75cb719d4a1181d0d27104544cdc" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 35997 } } peers { permanent_uuid: "dc9b942bcd6349a6adb9faaff24b94b3" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 41925 } }
I20250903 08:07:49.790474 14745 raft_consensus.cc:357] T 410b298ec531468680fbaa1fa3e1f7de P dc9b942bcd6349a6adb9faaff24b94b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34e13456e0c24db6a9b9b04a81897d08" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 35365 } } peers { permanent_uuid: "b9be75cb719d4a1181d0d27104544cdc" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 35997 } } peers { permanent_uuid: "dc9b942bcd6349a6adb9faaff24b94b3" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 41925 } }
I20250903 08:07:49.790620 14744 raft_consensus.cc:383] T 410b298ec531468680fbaa1fa3e1f7de P 34e13456e0c24db6a9b9b04a81897d08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:07:49.790628 14745 raft_consensus.cc:383] T 410b298ec531468680fbaa1fa3e1f7de P dc9b942bcd6349a6adb9faaff24b94b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:07:49.790686 14745 raft_consensus.cc:738] T 410b298ec531468680fbaa1fa3e1f7de P dc9b942bcd6349a6adb9faaff24b94b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc9b942bcd6349a6adb9faaff24b94b3, State: Initialized, Role: FOLLOWER
I20250903 08:07:49.790705 14744 raft_consensus.cc:738] T 410b298ec531468680fbaa1fa3e1f7de P 34e13456e0c24db6a9b9b04a81897d08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34e13456e0c24db6a9b9b04a81897d08, State: Initialized, Role: FOLLOWER
I20250903 08:07:49.790611 14743 raft_consensus.cc:357] T 410b298ec531468680fbaa1fa3e1f7de P b9be75cb719d4a1181d0d27104544cdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34e13456e0c24db6a9b9b04a81897d08" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 35365 } } peers { permanent_uuid: "b9be75cb719d4a1181d0d27104544cdc" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 35997 } } peers { permanent_uuid: "dc9b942bcd6349a6adb9faaff24b94b3" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 41925 } }
I20250903 08:07:49.790839 14743 raft_consensus.cc:383] T 410b298ec531468680fbaa1fa3e1f7de P b9be75cb719d4a1181d0d27104544cdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:07:49.790897 14743 raft_consensus.cc:738] T 410b298ec531468680fbaa1fa3e1f7de P b9be75cb719d4a1181d0d27104544cdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9be75cb719d4a1181d0d27104544cdc, State: Initialized, Role: FOLLOWER
I20250903 08:07:49.791183 14745 consensus_queue.cc:260] T 410b298ec531468680fbaa1fa3e1f7de P dc9b942bcd6349a6adb9faaff24b94b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34e13456e0c24db6a9b9b04a81897d08" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 35365 } } peers { permanent_uuid: "b9be75cb719d4a1181d0d27104544cdc" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 35997 } } peers { permanent_uuid: "dc9b942bcd6349a6adb9faaff24b94b3" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 41925 } }
I20250903 08:07:49.791209 14744 consensus_queue.cc:260] T 410b298ec531468680fbaa1fa3e1f7de P 34e13456e0c24db6a9b9b04a81897d08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34e13456e0c24db6a9b9b04a81897d08" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 35365 } } peers { permanent_uuid: "b9be75cb719d4a1181d0d27104544cdc" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 35997 } } peers { permanent_uuid: "dc9b942bcd6349a6adb9faaff24b94b3" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 41925 } }
I20250903 08:07:49.791273 14743 consensus_queue.cc:260] T 410b298ec531468680fbaa1fa3e1f7de P b9be75cb719d4a1181d0d27104544cdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34e13456e0c24db6a9b9b04a81897d08" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 35365 } } peers { permanent_uuid: "b9be75cb719d4a1181d0d27104544cdc" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 35997 } } peers { permanent_uuid: "dc9b942bcd6349a6adb9faaff24b94b3" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 41925 } }
I20250903 08:07:49.791826 14436 heartbeater.cc:499] Master 127.12.201.254:46669 was elected leader, sending a full tablet report...
I20250903 08:07:49.792012 14745 ts_tablet_manager.cc:1428] T 410b298ec531468680fbaa1fa3e1f7de P dc9b942bcd6349a6adb9faaff24b94b3: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250903 08:07:49.792054 14744 ts_tablet_manager.cc:1428] T 410b298ec531468680fbaa1fa3e1f7de P 34e13456e0c24db6a9b9b04a81897d08: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20250903 08:07:49.792438 14720 heartbeater.cc:499] Master 127.12.201.254:46669 was elected leader, sending a full tablet report...
I20250903 08:07:49.792472 14743 ts_tablet_manager.cc:1428] T 410b298ec531468680fbaa1fa3e1f7de P b9be75cb719d4a1181d0d27104544cdc: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20250903 08:07:49.792775 14578 heartbeater.cc:499] Master 127.12.201.254:46669 was elected leader, sending a full tablet report...
I20250903 08:07:49.916764 14749 raft_consensus.cc:491] T 410b298ec531468680fbaa1fa3e1f7de P dc9b942bcd6349a6adb9faaff24b94b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 08:07:49.916989 14749 raft_consensus.cc:513] T 410b298ec531468680fbaa1fa3e1f7de P dc9b942bcd6349a6adb9faaff24b94b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34e13456e0c24db6a9b9b04a81897d08" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 35365 } } peers { permanent_uuid: "b9be75cb719d4a1181d0d27104544cdc" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 35997 } } peers { permanent_uuid: "dc9b942bcd6349a6adb9faaff24b94b3" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 41925 } }
I20250903 08:07:49.918164 14749 leader_election.cc:290] T 410b298ec531468680fbaa1fa3e1f7de P dc9b942bcd6349a6adb9faaff24b94b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 34e13456e0c24db6a9b9b04a81897d08 (127.12.201.195:35365), b9be75cb719d4a1181d0d27104544cdc (127.12.201.194:35997)
I20250903 08:07:49.921499 14752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:49.918406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.194:35997 (local address 127.12.201.193:49001)
0903 08:07:49.918738 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0903 08:07:49.918749 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0903 08:07:49.918830 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:07:49.919111 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:07:49.919113 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:07:49.919155 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 08:07:49.919478 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:49.919490 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:49.920296 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:07:49.920298 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:07:49.921033 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:49.921041 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:49.921159 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:49.921182 (+ 23us) client_negotiation.cc:769] Sending connection context
0903 08:07:49.921259 (+ 77us) client_negotiation.cc:240] Negotiation successful
0903 08:07:49.921327 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"spinlock_wait_cycles":18048,"thread_start_us":81,"threads_started":1}
I20250903 08:07:49.921492 14751 raft_consensus.cc:491] T 410b298ec531468680fbaa1fa3e1f7de P b9be75cb719d4a1181d0d27104544cdc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 08:07:49.921689 14751 raft_consensus.cc:513] T 410b298ec531468680fbaa1fa3e1f7de P b9be75cb719d4a1181d0d27104544cdc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34e13456e0c24db6a9b9b04a81897d08" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 35365 } } peers { permanent_uuid: "b9be75cb719d4a1181d0d27104544cdc" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 35997 } } peers { permanent_uuid: "dc9b942bcd6349a6adb9faaff24b94b3" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 41925 } }
I20250903 08:07:49.921908 14742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:49.918525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.193:49001 (local address 127.12.201.194:35997)
0903 08:07:49.918658 (+ 133us) server_negotiation.cc:207] Beginning negotiation
0903 08:07:49.918663 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 08:07:49.918812 (+ 149us) server_negotiation.cc:408] Connection header received
0903 08:07:49.918940 (+ 128us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:07:49.918943 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:07:49.919035 (+ 92us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:07:49.919108 (+ 73us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 08:07:49.919617 (+ 509us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:49.920192 (+ 575us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:07:49.921172 (+ 980us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:49.921660 (+ 488us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:49.921701 (+ 41us) server_negotiation.cc:1036] Waiting for connection context
0903 08:07:49.921753 (+ 52us) server_negotiation.cc:300] Negotiation successful
0903 08:07:49.921810 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20250903 08:07:49.922494 14531 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "410b298ec531468680fbaa1fa3e1f7de" candidate_uuid: "dc9b942bcd6349a6adb9faaff24b94b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9be75cb719d4a1181d0d27104544cdc" is_pre_election: true
I20250903 08:07:49.922595 14753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:49.918381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.195:35365 (local address 127.12.201.193:32801)
0903 08:07:49.918658 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0903 08:07:49.918674 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0903 08:07:49.918780 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:07:49.919103 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:07:49.919106 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:07:49.919154 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 08:07:49.919920 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:49.919927 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:49.920716 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:07:49.920720 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:07:49.921558 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:49.921569 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:49.922300 (+ 731us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:49.922315 (+ 15us) client_negotiation.cc:769] Sending connection context
0903 08:07:49.922386 (+ 71us) client_negotiation.cc:240] Negotiation successful
0903 08:07:49.922458 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"spinlock_wait_cycles":28672,"thread_start_us":97,"threads_started":1}
I20250903 08:07:49.922627 14739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:49.918469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.193:32801 (local address 127.12.201.195:35365)
0903 08:07:49.918706 (+ 237us) server_negotiation.cc:207] Beginning negotiation
0903 08:07:49.918710 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:07:49.918723 (+ 13us) server_negotiation.cc:408] Connection header received
0903 08:07:49.918904 (+ 181us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:07:49.918908 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:07:49.918965 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:07:49.919073 (+ 108us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 08:07:49.920039 (+ 966us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:49.920605 (+ 566us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:07:49.921708 (+ 1103us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:49.922184 (+ 476us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:49.922220 (+ 36us) server_negotiation.cc:1036] Waiting for connection context
0903 08:07:49.922442 (+ 222us) server_negotiation.cc:300] Negotiation successful
0903 08:07:49.922515 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
I20250903 08:07:49.922685 14531 raft_consensus.cc:2466] T 410b298ec531468680fbaa1fa3e1f7de P b9be75cb719d4a1181d0d27104544cdc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc9b942bcd6349a6adb9faaff24b94b3 in term 0.
I20250903 08:07:49.922942 14751 leader_election.cc:290] T 410b298ec531468680fbaa1fa3e1f7de P b9be75cb719d4a1181d0d27104544cdc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 34e13456e0c24db6a9b9b04a81897d08 (127.12.201.195:35365), dc9b942bcd6349a6adb9faaff24b94b3 (127.12.201.193:41925)
I20250903 08:07:49.923501 14674 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "410b298ec531468680fbaa1fa3e1f7de" candidate_uuid: "dc9b942bcd6349a6adb9faaff24b94b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34e13456e0c24db6a9b9b04a81897d08" is_pre_election: true
I20250903 08:07:49.923714 14324 leader_election.cc:304] T 410b298ec531468680fbaa1fa3e1f7de P dc9b942bcd6349a6adb9faaff24b94b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9be75cb719d4a1181d0d27104544cdc, dc9b942bcd6349a6adb9faaff24b94b3; no voters:
I20250903 08:07:49.923774 14674 raft_consensus.cc:2466] T 410b298ec531468680fbaa1fa3e1f7de P 34e13456e0c24db6a9b9b04a81897d08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc9b942bcd6349a6adb9faaff24b94b3 in term 0.
I20250903 08:07:49.924015 14749 raft_consensus.cc:2802] T 410b298ec531468680fbaa1fa3e1f7de P dc9b942bcd6349a6adb9faaff24b94b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 08:07:49.924098 14749 raft_consensus.cc:491] T 410b298ec531468680fbaa1fa3e1f7de P dc9b942bcd6349a6adb9faaff24b94b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 08:07:49.924144 14749 raft_consensus.cc:3058] T 410b298ec531468680fbaa1fa3e1f7de P dc9b942bcd6349a6adb9faaff24b94b3 [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:07:49.925457 14749 raft_consensus.cc:513] T 410b298ec531468680fbaa1fa3e1f7de P dc9b942bcd6349a6adb9faaff24b94b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34e13456e0c24db6a9b9b04a81897d08" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 35365 } } peers { permanent_uuid: "b9be75cb719d4a1181d0d27104544cdc" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 35997 } } peers { permanent_uuid: "dc9b942bcd6349a6adb9faaff24b94b3" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 41925 } }
I20250903 08:07:49.925756 14471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:49.923133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.195:35365 (local address 127.12.201.194:59069)
0903 08:07:49.923358 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0903 08:07:49.923391 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0903 08:07:49.923468 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:07:49.923761 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:07:49.923765 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:07:49.923785 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 08:07:49.924007 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:49.924015 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:49.924796 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:07:49.924801 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:07:49.925390 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:49.925398 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:49.925506 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:49.925526 (+ 20us) client_negotiation.cc:769] Sending connection context
0903 08:07:49.925578 (+ 52us) client_negotiation.cc:240] Negotiation successful
0903 08:07:49.925638 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140}
I20250903 08:07:49.925823 14749 leader_election.cc:290] T 410b298ec531468680fbaa1fa3e1f7de P dc9b942bcd6349a6adb9faaff24b94b3 [CANDIDATE]: Term 1 election: Requested vote from peers 34e13456e0c24db6a9b9b04a81897d08 (127.12.201.195:35365), b9be75cb719d4a1181d0d27104544cdc (127.12.201.194:35997)
I20250903 08:07:49.926200 14739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:49.923358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.194:59069 (local address 127.12.201.195:35365)
0903 08:07:49.923463 (+ 105us) server_negotiation.cc:207] Beginning negotiation
0903 08:07:49.923480 (+ 17us) server_negotiation.cc:400] Waiting for connection header
0903 08:07:49.923496 (+ 16us) server_negotiation.cc:408] Connection header received
0903 08:07:49.923538 (+ 42us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:07:49.923541 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:07:49.923580 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:07:49.923664 (+ 84us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 08:07:49.924136 (+ 472us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:49.924678 (+ 542us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:07:49.925512 (+ 834us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:49.925953 (+ 441us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:49.925993 (+ 40us) server_negotiation.cc:1036] Waiting for connection context
0903 08:07:49.926038 (+ 45us) server_negotiation.cc:300] Negotiation successful
0903 08:07:49.926091 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20250903 08:07:49.926308 14531 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "410b298ec531468680fbaa1fa3e1f7de" candidate_uuid: "dc9b942bcd6349a6adb9faaff24b94b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9be75cb719d4a1181d0d27104544cdc"
I20250903 08:07:49.926420 14674 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "410b298ec531468680fbaa1fa3e1f7de" candidate_uuid: "dc9b942bcd6349a6adb9faaff24b94b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34e13456e0c24db6a9b9b04a81897d08"
I20250903 08:07:49.926445 14531 raft_consensus.cc:3058] T 410b298ec531468680fbaa1fa3e1f7de P b9be75cb719d4a1181d0d27104544cdc [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:07:49.926553 14674 raft_consensus.cc:3058] T 410b298ec531468680fbaa1fa3e1f7de P 34e13456e0c24db6a9b9b04a81897d08 [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:07:49.926723 14673 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "410b298ec531468680fbaa1fa3e1f7de" candidate_uuid: "b9be75cb719d4a1181d0d27104544cdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34e13456e0c24db6a9b9b04a81897d08" is_pre_election: true
I20250903 08:07:49.927678 14741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:49.923567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.194:59923 (local address 127.12.201.193:41925)
0903 08:07:49.923684 (+ 117us) server_negotiation.cc:207] Beginning negotiation
0903 08:07:49.923688 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:07:49.923934 (+ 246us) server_negotiation.cc:408] Connection header received
0903 08:07:49.924129 (+ 195us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:07:49.924134 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:07:49.924187 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:07:49.924237 (+ 50us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 08:07:49.924807 (+ 570us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:49.925495 (+ 688us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:07:49.926685 (+ 1190us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:49.927185 (+ 500us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:49.927222 (+ 37us) server_negotiation.cc:1036] Waiting for connection context
0903 08:07:49.927500 (+ 278us) server_negotiation.cc:300] Negotiation successful
0903 08:07:49.927574 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20250903 08:07:49.927711 14531 raft_consensus.cc:2466] T 410b298ec531468680fbaa1fa3e1f7de P b9be75cb719d4a1181d0d27104544cdc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc9b942bcd6349a6adb9faaff24b94b3 in term 1.
I20250903 08:07:49.927870 14674 raft_consensus.cc:2466] T 410b298ec531468680fbaa1fa3e1f7de P 34e13456e0c24db6a9b9b04a81897d08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc9b942bcd6349a6adb9faaff24b94b3 in term 1.
I20250903 08:07:49.928026 14324 leader_election.cc:304] T 410b298ec531468680fbaa1fa3e1f7de P dc9b942bcd6349a6adb9faaff24b94b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9be75cb719d4a1181d0d27104544cdc, dc9b942bcd6349a6adb9faaff24b94b3; no voters:
I20250903 08:07:49.928087 14754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:49.923239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.193:41925 (local address 127.12.201.194:59923)
0903 08:07:49.923864 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0903 08:07:49.923877 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0903 08:07:49.923998 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:07:49.924369 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:07:49.924372 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:07:49.924393 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 08:07:49.924677 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:49.924685 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:49.925632 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:07:49.925635 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:07:49.926543 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:49.926554 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:49.927315 (+ 761us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:49.927335 (+ 20us) client_negotiation.cc:769] Sending connection context
0903 08:07:49.927862 (+ 527us) client_negotiation.cc:240] Negotiation successful
0903 08:07:49.927930 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":133,"threads_started":1}
I20250903 08:07:49.928205 14749 raft_consensus.cc:2802] T 410b298ec531468680fbaa1fa3e1f7de P dc9b942bcd6349a6adb9faaff24b94b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 08:07:49.928706 14390 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "410b298ec531468680fbaa1fa3e1f7de" candidate_uuid: "b9be75cb719d4a1181d0d27104544cdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc9b942bcd6349a6adb9faaff24b94b3" is_pre_election: true
I20250903 08:07:49.929056 14749 raft_consensus.cc:695] T 410b298ec531468680fbaa1fa3e1f7de P dc9b942bcd6349a6adb9faaff24b94b3 [term 1 LEADER]: Becoming Leader. State: Replica: dc9b942bcd6349a6adb9faaff24b94b3, State: Running, Role: LEADER
I20250903 08:07:49.929399 14749 consensus_queue.cc:237] T 410b298ec531468680fbaa1fa3e1f7de P dc9b942bcd6349a6adb9faaff24b94b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34e13456e0c24db6a9b9b04a81897d08" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 35365 } } peers { permanent_uuid: "b9be75cb719d4a1181d0d27104544cdc" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 35997 } } peers { permanent_uuid: "dc9b942bcd6349a6adb9faaff24b94b3" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 41925 } }
I20250903 08:07:49.932073 14465 leader_election.cc:304] T 410b298ec531468680fbaa1fa3e1f7de P b9be75cb719d4a1181d0d27104544cdc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9be75cb719d4a1181d0d27104544cdc; no voters: 34e13456e0c24db6a9b9b04a81897d08, dc9b942bcd6349a6adb9faaff24b94b3
I20250903 08:07:49.932333 14751 raft_consensus.cc:2747] T 410b298ec531468680fbaa1fa3e1f7de P b9be75cb719d4a1181d0d27104544cdc [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 08:07:49.932809 14229 catalog_manager.cc:5582] T 410b298ec531468680fbaa1fa3e1f7de P dc9b942bcd6349a6adb9faaff24b94b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to dc9b942bcd6349a6adb9faaff24b94b3 (127.12.201.193). New cstate: current_term: 1 leader_uuid: "dc9b942bcd6349a6adb9faaff24b94b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34e13456e0c24db6a9b9b04a81897d08" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 35365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9be75cb719d4a1181d0d27104544cdc" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 35997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc9b942bcd6349a6adb9faaff24b94b3" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 41925 } health_report { overall_health: HEALTHY } } }
I20250903 08:07:49.966812 14741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:49.963013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49954 (local address 127.12.201.193:41925)
0903 08:07:49.963152 (+ 139us) server_negotiation.cc:207] Beginning negotiation
0903 08:07:49.963157 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 08:07:49.963176 (+ 19us) server_negotiation.cc:408] Connection header received
0903 08:07:49.963284 (+ 108us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:07:49.963288 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:07:49.963366 (+ 78us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:07:49.963477 (+ 111us) server_negotiation.cc:227] Negotiated authn=TOKEN
0903 08:07:49.963930 (+ 453us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:49.964682 (+ 752us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:07:49.965426 (+ 744us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:49.965632 (+ 206us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:49.965706 (+ 74us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0903 08:07:49.966298 (+ 592us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0903 08:07:49.966423 (+ 125us) server_negotiation.cc:1036] Waiting for connection context
0903 08:07:49.966597 (+ 174us) server_negotiation.cc:300] Negotiation successful
0903 08:07:49.966667 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20250903 08:07:49.973639 14531 raft_consensus.cc:1273] T 410b298ec531468680fbaa1fa3e1f7de P b9be75cb719d4a1181d0d27104544cdc [term 1 FOLLOWER]: Refusing update from remote peer dc9b942bcd6349a6adb9faaff24b94b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 08:07:49.973779 14674 raft_consensus.cc:1273] T 410b298ec531468680fbaa1fa3e1f7de P 34e13456e0c24db6a9b9b04a81897d08 [term 1 FOLLOWER]: Refusing update from remote peer dc9b942bcd6349a6adb9faaff24b94b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 08:07:49.974263 14755 consensus_queue.cc:1035] T 410b298ec531468680fbaa1fa3e1f7de P dc9b942bcd6349a6adb9faaff24b94b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9be75cb719d4a1181d0d27104544cdc" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 35997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250903 08:07:49.975144 14721 tablet.cc:2378] T 410b298ec531468680fbaa1fa3e1f7de P 34e13456e0c24db6a9b9b04a81897d08: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250903 08:07:49.975350 14755 consensus_queue.cc:1035] T 410b298ec531468680fbaa1fa3e1f7de P dc9b942bcd6349a6adb9faaff24b94b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34e13456e0c24db6a9b9b04a81897d08" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 35365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 08:07:49.983168 14759 mvcc.cc:204] Tried to move back new op lower bound from 7196208619404402688 to 7196208619233431552. Current Snapshot: MvccSnapshot[applied={T|T < 7196208619404402688}]
I20250903 08:07:49.983728 14760 mvcc.cc:204] Tried to move back new op lower bound from 7196208619404402688 to 7196208619233431552. Current Snapshot: MvccSnapshot[applied={T|T < 7196208619404402688}]
I20250903 08:07:49.987732 14758 mvcc.cc:204] Tried to move back new op lower bound from 7196208619404402688 to 7196208619233431552. Current Snapshot: MvccSnapshot[applied={T|T < 7196208619404402688 or (T in {7196208619404402688})}]
I20250903 08:07:49.993943 14229 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52336:
table { table_name: "test-table" } modify_external_catalogs: true
I20250903 08:07:49.994127 14229 catalog_manager.cc:2730] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52336:
table { table_name: "test-table" } modify_external_catalogs: true
I20250903 08:07:49.997326 14229 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 5b996e3f77d84707b4bfc52734aeb4c9: Sending DeleteTablet for 3 replicas of tablet 410b298ec531468680fbaa1fa3e1f7de
I20250903 08:07:49.998148 14370 tablet_service.cc:1515] Processing DeleteTablet for tablet 410b298ec531468680fbaa1fa3e1f7de with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 08:07:49 UTC) from {username='kudu', principal='kudu/127.12.201.254@KRBTEST.COM'} at 127.0.0.1:49938
I20250903 08:07:49.998262 14654 tablet_service.cc:1515] Processing DeleteTablet for tablet 410b298ec531468680fbaa1fa3e1f7de with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 08:07:49 UTC) from {username='kudu', principal='kudu/127.12.201.254@KRBTEST.COM'} at 127.0.0.1:45306
I20250903 08:07:49.998468 14512 tablet_service.cc:1515] Processing DeleteTablet for tablet 410b298ec531468680fbaa1fa3e1f7de with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 08:07:49 UTC) from {username='kudu', principal='kudu/127.12.201.254@KRBTEST.COM'} at 127.0.0.1:34316
I20250903 08:07:49.998821 14768 tablet_replica.cc:331] T 410b298ec531468680fbaa1fa3e1f7de P dc9b942bcd6349a6adb9faaff24b94b3: stopping tablet replica
I20250903 08:07:49.998888 14770 tablet_replica.cc:331] T 410b298ec531468680fbaa1fa3e1f7de P b9be75cb719d4a1181d0d27104544cdc: stopping tablet replica
I20250903 08:07:49.999154 14768 raft_consensus.cc:2241] T 410b298ec531468680fbaa1fa3e1f7de P dc9b942bcd6349a6adb9faaff24b94b3 [term 1 LEADER]: Raft consensus shutting down.
I20250903 08:07:49.999154 14770 raft_consensus.cc:2241] T 410b298ec531468680fbaa1fa3e1f7de P b9be75cb719d4a1181d0d27104544cdc [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 08:07:49.999384 14229 catalog_manager.cc:2232] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52336:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 08:07:49.999480 14770 raft_consensus.cc:2270] T 410b298ec531468680fbaa1fa3e1f7de P b9be75cb719d4a1181d0d27104544cdc [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 08:07:49.999763 14768 raft_consensus.cc:2270] T 410b298ec531468680fbaa1fa3e1f7de P dc9b942bcd6349a6adb9faaff24b94b3 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250903 08:07:49.999812 14229 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 08:07:50.000948 14768 ts_tablet_manager.cc:1905] T 410b298ec531468680fbaa1fa3e1f7de P dc9b942bcd6349a6adb9faaff24b94b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 08:07:50.000948 14770 ts_tablet_manager.cc:1905] T 410b298ec531468680fbaa1fa3e1f7de P b9be75cb719d4a1181d0d27104544cdc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 08:07:50.001459 14769 tablet_replica.cc:331] T 410b298ec531468680fbaa1fa3e1f7de P 34e13456e0c24db6a9b9b04a81897d08: stopping tablet replica
I20250903 08:07:50.001854 14769 raft_consensus.cc:2241] T 410b298ec531468680fbaa1fa3e1f7de P 34e13456e0c24db6a9b9b04a81897d08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 08:07:50.002152 14769 raft_consensus.cc:2270] T 410b298ec531468680fbaa1fa3e1f7de P 34e13456e0c24db6a9b9b04a81897d08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 08:07:50.004149 14768 ts_tablet_manager.cc:1918] T 410b298ec531468680fbaa1fa3e1f7de P dc9b942bcd6349a6adb9faaff24b94b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250903 08:07:50.004292 14768 log.cc:1199] T 410b298ec531468680fbaa1fa3e1f7de P dc9b942bcd6349a6adb9faaff24b94b3: Deleting WAL directory at /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/minicluster-data/ts-0/wal/wals/410b298ec531468680fbaa1fa3e1f7de
I20250903 08:07:50.004691 14768 ts_tablet_manager.cc:1939] T 410b298ec531468680fbaa1fa3e1f7de P dc9b942bcd6349a6adb9faaff24b94b3: Deleting consensus metadata
I20250903 08:07:50.005908 14217 catalog_manager.cc:4928] TS dc9b942bcd6349a6adb9faaff24b94b3 (127.12.201.193:41925): tablet 410b298ec531468680fbaa1fa3e1f7de (table test-table [id=e9907ae074a24f61a5c1417dc679ce7e]) successfully deleted
I20250903 08:07:50.006549 14370 tablet_service.cc:1468] Processing CreateTablet for tablet 4ab1cbf6753a46aeb78c5783e0e2da6c (DEFAULT_TABLE table=test-table [id=82c1319f7a594578a286a84b3f4677a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:07:50.006734 14512 tablet_service.cc:1468] Processing CreateTablet for tablet 4ab1cbf6753a46aeb78c5783e0e2da6c (DEFAULT_TABLE table=test-table [id=82c1319f7a594578a286a84b3f4677a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:07:50.006795 14370 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ab1cbf6753a46aeb78c5783e0e2da6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:07:50.007634 14654 tablet_service.cc:1468] Processing CreateTablet for tablet 4ab1cbf6753a46aeb78c5783e0e2da6c (DEFAULT_TABLE table=test-table [id=82c1319f7a594578a286a84b3f4677a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:07:50.007895 14654 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ab1cbf6753a46aeb78c5783e0e2da6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:07:50.009027 14745 tablet_bootstrap.cc:492] T 4ab1cbf6753a46aeb78c5783e0e2da6c P dc9b942bcd6349a6adb9faaff24b94b3: Bootstrap starting.
I20250903 08:07:50.010097 14745 tablet_bootstrap.cc:654] T 4ab1cbf6753a46aeb78c5783e0e2da6c P dc9b942bcd6349a6adb9faaff24b94b3: Neither blocks nor log segments found. Creating new log.
I20250903 08:07:50.010291 14744 tablet_bootstrap.cc:492] T 4ab1cbf6753a46aeb78c5783e0e2da6c P 34e13456e0c24db6a9b9b04a81897d08: Bootstrap starting.
I20250903 08:07:50.006965 14512 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ab1cbf6753a46aeb78c5783e0e2da6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:07:50.011354 14744 tablet_bootstrap.cc:654] T 4ab1cbf6753a46aeb78c5783e0e2da6c P 34e13456e0c24db6a9b9b04a81897d08: Neither blocks nor log segments found. Creating new log.
I20250903 08:07:50.014197 14743 tablet_bootstrap.cc:492] T 4ab1cbf6753a46aeb78c5783e0e2da6c P b9be75cb719d4a1181d0d27104544cdc: Bootstrap starting.
I20250903 08:07:50.015395 14743 tablet_bootstrap.cc:654] T 4ab1cbf6753a46aeb78c5783e0e2da6c P b9be75cb719d4a1181d0d27104544cdc: Neither blocks nor log segments found. Creating new log.
I20250903 08:07:50.016630 14743 tablet_bootstrap.cc:492] T 4ab1cbf6753a46aeb78c5783e0e2da6c P b9be75cb719d4a1181d0d27104544cdc: No bootstrap required, opened a new log
I20250903 08:07:50.016717 14743 ts_tablet_manager.cc:1397] T 4ab1cbf6753a46aeb78c5783e0e2da6c P b9be75cb719d4a1181d0d27104544cdc: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250903 08:07:50.017223 14743 raft_consensus.cc:357] T 4ab1cbf6753a46aeb78c5783e0e2da6c P b9be75cb719d4a1181d0d27104544cdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9b942bcd6349a6adb9faaff24b94b3" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 41925 } } peers { permanent_uuid: "b9be75cb719d4a1181d0d27104544cdc" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 35997 } } peers { permanent_uuid: "34e13456e0c24db6a9b9b04a81897d08" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 35365 } }
I20250903 08:07:50.017336 14743 raft_consensus.cc:383] T 4ab1cbf6753a46aeb78c5783e0e2da6c P b9be75cb719d4a1181d0d27104544cdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:07:50.017374 14743 raft_consensus.cc:738] T 4ab1cbf6753a46aeb78c5783e0e2da6c P b9be75cb719d4a1181d0d27104544cdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9be75cb719d4a1181d0d27104544cdc, State: Initialized, Role: FOLLOWER
I20250903 08:07:50.017498 14743 consensus_queue.cc:260] T 4ab1cbf6753a46aeb78c5783e0e2da6c P b9be75cb719d4a1181d0d27104544cdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9b942bcd6349a6adb9faaff24b94b3" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 41925 } } peers { permanent_uuid: "b9be75cb719d4a1181d0d27104544cdc" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 35997 } } peers { permanent_uuid: "34e13456e0c24db6a9b9b04a81897d08" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 35365 } }
I20250903 08:07:50.017767 14743 ts_tablet_manager.cc:1428] T 4ab1cbf6753a46aeb78c5783e0e2da6c P b9be75cb719d4a1181d0d27104544cdc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250903 08:07:50.018466 14770 ts_tablet_manager.cc:1918] T 410b298ec531468680fbaa1fa3e1f7de P b9be75cb719d4a1181d0d27104544cdc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250903 08:07:50.018553 14770 log.cc:1199] T 410b298ec531468680fbaa1fa3e1f7de P b9be75cb719d4a1181d0d27104544cdc: Deleting WAL directory at /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/minicluster-data/ts-1/wal/wals/410b298ec531468680fbaa1fa3e1f7de
I20250903 08:07:50.018855 14770 ts_tablet_manager.cc:1939] T 410b298ec531468680fbaa1fa3e1f7de P b9be75cb719d4a1181d0d27104544cdc: Deleting consensus metadata
I20250903 08:07:50.019071 14769 ts_tablet_manager.cc:1905] T 410b298ec531468680fbaa1fa3e1f7de P 34e13456e0c24db6a9b9b04a81897d08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 08:07:50.019997 14218 catalog_manager.cc:4928] TS b9be75cb719d4a1181d0d27104544cdc (127.12.201.194:35997): tablet 410b298ec531468680fbaa1fa3e1f7de (table test-table [id=e9907ae074a24f61a5c1417dc679ce7e]) successfully deleted
I20250903 08:07:50.023695 14744 tablet_bootstrap.cc:492] T 4ab1cbf6753a46aeb78c5783e0e2da6c P 34e13456e0c24db6a9b9b04a81897d08: No bootstrap required, opened a new log
I20250903 08:07:50.023789 14744 ts_tablet_manager.cc:1397] T 4ab1cbf6753a46aeb78c5783e0e2da6c P 34e13456e0c24db6a9b9b04a81897d08: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20250903 08:07:50.024190 14744 raft_consensus.cc:357] T 4ab1cbf6753a46aeb78c5783e0e2da6c P 34e13456e0c24db6a9b9b04a81897d08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9b942bcd6349a6adb9faaff24b94b3" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 41925 } } peers { permanent_uuid: "b9be75cb719d4a1181d0d27104544cdc" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 35997 } } peers { permanent_uuid: "34e13456e0c24db6a9b9b04a81897d08" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 35365 } }
I20250903 08:07:50.024315 14744 raft_consensus.cc:383] T 4ab1cbf6753a46aeb78c5783e0e2da6c P 34e13456e0c24db6a9b9b04a81897d08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:07:50.024356 14744 raft_consensus.cc:738] T 4ab1cbf6753a46aeb78c5783e0e2da6c P 34e13456e0c24db6a9b9b04a81897d08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34e13456e0c24db6a9b9b04a81897d08, State: Initialized, Role: FOLLOWER
I20250903 08:07:50.024518 14744 consensus_queue.cc:260] T 4ab1cbf6753a46aeb78c5783e0e2da6c P 34e13456e0c24db6a9b9b04a81897d08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9b942bcd6349a6adb9faaff24b94b3" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 41925 } } peers { permanent_uuid: "b9be75cb719d4a1181d0d27104544cdc" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 35997 } } peers { permanent_uuid: "34e13456e0c24db6a9b9b04a81897d08" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 35365 } }
I20250903 08:07:50.024752 14744 ts_tablet_manager.cc:1428] T 4ab1cbf6753a46aeb78c5783e0e2da6c P 34e13456e0c24db6a9b9b04a81897d08: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250903 08:07:50.026674 14769 ts_tablet_manager.cc:1918] T 410b298ec531468680fbaa1fa3e1f7de P 34e13456e0c24db6a9b9b04a81897d08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250903 08:07:50.026762 14769 log.cc:1199] T 410b298ec531468680fbaa1fa3e1f7de P 34e13456e0c24db6a9b9b04a81897d08: Deleting WAL directory at /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/minicluster-data/ts-2/wal/wals/410b298ec531468680fbaa1fa3e1f7de
I20250903 08:07:50.027127 14769 ts_tablet_manager.cc:1939] T 410b298ec531468680fbaa1fa3e1f7de P 34e13456e0c24db6a9b9b04a81897d08: Deleting consensus metadata
I20250903 08:07:50.028312 14216 catalog_manager.cc:4928] TS 34e13456e0c24db6a9b9b04a81897d08 (127.12.201.195:35365): tablet 410b298ec531468680fbaa1fa3e1f7de (table test-table [id=e9907ae074a24f61a5c1417dc679ce7e]) successfully deleted
I20250903 08:07:50.029178 14745 tablet_bootstrap.cc:492] T 4ab1cbf6753a46aeb78c5783e0e2da6c P dc9b942bcd6349a6adb9faaff24b94b3: No bootstrap required, opened a new log
I20250903 08:07:50.029289 14745 ts_tablet_manager.cc:1397] T 4ab1cbf6753a46aeb78c5783e0e2da6c P dc9b942bcd6349a6adb9faaff24b94b3: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.000s
I20250903 08:07:50.029814 14745 raft_consensus.cc:357] T 4ab1cbf6753a46aeb78c5783e0e2da6c P dc9b942bcd6349a6adb9faaff24b94b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9b942bcd6349a6adb9faaff24b94b3" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 41925 } } peers { permanent_uuid: "b9be75cb719d4a1181d0d27104544cdc" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 35997 } } peers { permanent_uuid: "34e13456e0c24db6a9b9b04a81897d08" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 35365 } }
I20250903 08:07:50.029912 14745 raft_consensus.cc:383] T 4ab1cbf6753a46aeb78c5783e0e2da6c P dc9b942bcd6349a6adb9faaff24b94b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:07:50.029954 14745 raft_consensus.cc:738] T 4ab1cbf6753a46aeb78c5783e0e2da6c P dc9b942bcd6349a6adb9faaff24b94b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc9b942bcd6349a6adb9faaff24b94b3, State: Initialized, Role: FOLLOWER
I20250903 08:07:50.030087 14745 consensus_queue.cc:260] T 4ab1cbf6753a46aeb78c5783e0e2da6c P dc9b942bcd6349a6adb9faaff24b94b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9b942bcd6349a6adb9faaff24b94b3" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 41925 } } peers { permanent_uuid: "b9be75cb719d4a1181d0d27104544cdc" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 35997 } } peers { permanent_uuid: "34e13456e0c24db6a9b9b04a81897d08" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 35365 } }
I20250903 08:07:50.030323 14745 ts_tablet_manager.cc:1428] T 4ab1cbf6753a46aeb78c5783e0e2da6c P dc9b942bcd6349a6adb9faaff24b94b3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20250903 08:07:50.035180 14437 tablet.cc:2378] T 4ab1cbf6753a46aeb78c5783e0e2da6c P dc9b942bcd6349a6adb9faaff24b94b3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250903 08:07:50.084632 14750 raft_consensus.cc:491] T 4ab1cbf6753a46aeb78c5783e0e2da6c P 34e13456e0c24db6a9b9b04a81897d08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 08:07:50.084792 14750 raft_consensus.cc:513] T 4ab1cbf6753a46aeb78c5783e0e2da6c P 34e13456e0c24db6a9b9b04a81897d08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9b942bcd6349a6adb9faaff24b94b3" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 41925 } } peers { permanent_uuid: "b9be75cb719d4a1181d0d27104544cdc" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 35997 } } peers { permanent_uuid: "34e13456e0c24db6a9b9b04a81897d08" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 35365 } }
I20250903 08:07:50.085894 14750 leader_election.cc:290] T 4ab1cbf6753a46aeb78c5783e0e2da6c P 34e13456e0c24db6a9b9b04a81897d08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc9b942bcd6349a6adb9faaff24b94b3 (127.12.201.193:41925), b9be75cb719d4a1181d0d27104544cdc (127.12.201.194:35997)
I20250903 08:07:50.089140 14774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:50.086156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.194:35997 (local address 127.12.201.195:57383)
0903 08:07:50.086557 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0903 08:07:50.086569 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0903 08:07:50.086637 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:07:50.086931 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:07:50.086934 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:07:50.086949 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 08:07:50.087189 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:50.087198 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:50.087901 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:07:50.087907 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:07:50.088720 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:50.088728 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:50.088836 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:50.088858 (+ 22us) client_negotiation.cc:769] Sending connection context
0903 08:07:50.088909 (+ 51us) client_negotiation.cc:240] Negotiation successful
0903 08:07:50.088977 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":76,"threads_started":1}
I20250903 08:07:50.089648 14742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:50.086523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.195:57383 (local address 127.12.201.194:35997)
0903 08:07:50.086646 (+ 123us) server_negotiation.cc:207] Beginning negotiation
0903 08:07:50.086651 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 08:07:50.086694 (+ 43us) server_negotiation.cc:408] Connection header received
0903 08:07:50.086764 (+ 70us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:07:50.086768 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:07:50.086823 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:07:50.086925 (+ 102us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 08:07:50.087319 (+ 394us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:50.087791 (+ 472us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:07:50.088846 (+ 1055us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:50.089343 (+ 497us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:50.089409 (+ 66us) server_negotiation.cc:1036] Waiting for connection context
0903 08:07:50.089463 (+ 54us) server_negotiation.cc:300] Negotiation successful
0903 08:07:50.089526 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20250903 08:07:50.089164 14613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:50.086089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.193:41925 (local address 127.12.201.195:37033)
0903 08:07:50.086302 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0903 08:07:50.086317 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0903 08:07:50.086442 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:07:50.086839 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:07:50.086841 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:07:50.086857 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 08:07:50.087136 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:50.087144 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:50.088124 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:07:50.088127 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:07:50.088887 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:50.088898 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:50.088983 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:50.088995 (+ 12us) client_negotiation.cc:769] Sending connection context
0903 08:07:50.089040 (+ 45us) client_negotiation.cc:240] Negotiation successful
0903 08:07:50.089080 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127}
I20250903 08:07:50.089998 14741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:50.086182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.195:37033 (local address 127.12.201.193:41925)
0903 08:07:50.086333 (+ 151us) server_negotiation.cc:207] Beginning negotiation
0903 08:07:50.086340 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0903 08:07:50.086379 (+ 39us) server_negotiation.cc:408] Connection header received
0903 08:07:50.086588 (+ 209us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:07:50.086595 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:07:50.086654 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:07:50.086724 (+ 70us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 08:07:50.087262 (+ 538us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:50.088009 (+ 747us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:07:50.089243 (+ 1234us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:50.089721 (+ 478us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:50.089770 (+ 49us) server_negotiation.cc:1036] Waiting for connection context
0903 08:07:50.089821 (+ 51us) server_negotiation.cc:300] Negotiation successful
0903 08:07:50.089877 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20250903 08:07:50.090026 14531 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ab1cbf6753a46aeb78c5783e0e2da6c" candidate_uuid: "34e13456e0c24db6a9b9b04a81897d08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9be75cb719d4a1181d0d27104544cdc" is_pre_election: true
I20250903 08:07:50.090179 14531 raft_consensus.cc:2466] T 4ab1cbf6753a46aeb78c5783e0e2da6c P b9be75cb719d4a1181d0d27104544cdc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34e13456e0c24db6a9b9b04a81897d08 in term 0.
I20250903 08:07:50.090504 14390 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ab1cbf6753a46aeb78c5783e0e2da6c" candidate_uuid: "34e13456e0c24db6a9b9b04a81897d08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc9b942bcd6349a6adb9faaff24b94b3" is_pre_election: true
I20250903 08:07:50.090543 14608 leader_election.cc:304] T 4ab1cbf6753a46aeb78c5783e0e2da6c P 34e13456e0c24db6a9b9b04a81897d08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34e13456e0c24db6a9b9b04a81897d08, b9be75cb719d4a1181d0d27104544cdc; no voters:
I20250903 08:07:50.090655 14390 raft_consensus.cc:2466] T 4ab1cbf6753a46aeb78c5783e0e2da6c P dc9b942bcd6349a6adb9faaff24b94b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34e13456e0c24db6a9b9b04a81897d08 in term 0.
I20250903 08:07:50.090777 14750 raft_consensus.cc:2802] T 4ab1cbf6753a46aeb78c5783e0e2da6c P 34e13456e0c24db6a9b9b04a81897d08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 08:07:50.090823 14750 raft_consensus.cc:491] T 4ab1cbf6753a46aeb78c5783e0e2da6c P 34e13456e0c24db6a9b9b04a81897d08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 08:07:50.090860 14750 raft_consensus.cc:3058] T 4ab1cbf6753a46aeb78c5783e0e2da6c P 34e13456e0c24db6a9b9b04a81897d08 [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:07:50.092042 14750 raft_consensus.cc:513] T 4ab1cbf6753a46aeb78c5783e0e2da6c P 34e13456e0c24db6a9b9b04a81897d08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9b942bcd6349a6adb9faaff24b94b3" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 41925 } } peers { permanent_uuid: "b9be75cb719d4a1181d0d27104544cdc" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 35997 } } peers { permanent_uuid: "34e13456e0c24db6a9b9b04a81897d08" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 35365 } }
I20250903 08:07:50.092418 14750 leader_election.cc:290] T 4ab1cbf6753a46aeb78c5783e0e2da6c P 34e13456e0c24db6a9b9b04a81897d08 [CANDIDATE]: Term 1 election: Requested vote from peers dc9b942bcd6349a6adb9faaff24b94b3 (127.12.201.193:41925), b9be75cb719d4a1181d0d27104544cdc (127.12.201.194:35997)
I20250903 08:07:50.092728 14531 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ab1cbf6753a46aeb78c5783e0e2da6c" candidate_uuid: "34e13456e0c24db6a9b9b04a81897d08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9be75cb719d4a1181d0d27104544cdc"
I20250903 08:07:50.092764 14390 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ab1cbf6753a46aeb78c5783e0e2da6c" candidate_uuid: "34e13456e0c24db6a9b9b04a81897d08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc9b942bcd6349a6adb9faaff24b94b3"
I20250903 08:07:50.092856 14531 raft_consensus.cc:3058] T 4ab1cbf6753a46aeb78c5783e0e2da6c P b9be75cb719d4a1181d0d27104544cdc [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:07:50.092872 14390 raft_consensus.cc:3058] T 4ab1cbf6753a46aeb78c5783e0e2da6c P dc9b942bcd6349a6adb9faaff24b94b3 [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:07:50.093736 14390 raft_consensus.cc:2466] T 4ab1cbf6753a46aeb78c5783e0e2da6c P dc9b942bcd6349a6adb9faaff24b94b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34e13456e0c24db6a9b9b04a81897d08 in term 1.
I20250903 08:07:50.093837 14531 raft_consensus.cc:2466] T 4ab1cbf6753a46aeb78c5783e0e2da6c P b9be75cb719d4a1181d0d27104544cdc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34e13456e0c24db6a9b9b04a81897d08 in term 1.
I20250903 08:07:50.094058 14607 leader_election.cc:304] T 4ab1cbf6753a46aeb78c5783e0e2da6c P 34e13456e0c24db6a9b9b04a81897d08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34e13456e0c24db6a9b9b04a81897d08, dc9b942bcd6349a6adb9faaff24b94b3; no voters:
I20250903 08:07:50.094265 14750 raft_consensus.cc:2802] T 4ab1cbf6753a46aeb78c5783e0e2da6c P 34e13456e0c24db6a9b9b04a81897d08 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 08:07:50.094492 14750 raft_consensus.cc:695] T 4ab1cbf6753a46aeb78c5783e0e2da6c P 34e13456e0c24db6a9b9b04a81897d08 [term 1 LEADER]: Becoming Leader. State: Replica: 34e13456e0c24db6a9b9b04a81897d08, State: Running, Role: LEADER
I20250903 08:07:50.094743 14750 consensus_queue.cc:237] T 4ab1cbf6753a46aeb78c5783e0e2da6c P 34e13456e0c24db6a9b9b04a81897d08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9b942bcd6349a6adb9faaff24b94b3" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 41925 } } peers { permanent_uuid: "b9be75cb719d4a1181d0d27104544cdc" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 35997 } } peers { permanent_uuid: "34e13456e0c24db6a9b9b04a81897d08" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 35365 } }
I20250903 08:07:50.097214 14230 catalog_manager.cc:5582] T 4ab1cbf6753a46aeb78c5783e0e2da6c P 34e13456e0c24db6a9b9b04a81897d08 reported cstate change: term changed from 0 to 1, leader changed from <none> to 34e13456e0c24db6a9b9b04a81897d08 (127.12.201.195). New cstate: current_term: 1 leader_uuid: "34e13456e0c24db6a9b9b04a81897d08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9b942bcd6349a6adb9faaff24b94b3" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 41925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9be75cb719d4a1181d0d27104544cdc" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 35997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "34e13456e0c24db6a9b9b04a81897d08" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 35365 } health_report { overall_health: HEALTHY } } }
Sep 03 08:07:50 dist-test-slave-27r9 krb5kdc[14183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886868, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.254@KRBTEST.COM for kudu/127.12.201.254@KRBTEST.COM
I20250903 08:07:50.141522 14280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:50.132886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52340 (local address 127.12.201.254:46669)
0903 08:07:50.133009 (+ 123us) server_negotiation.cc:207] Beginning negotiation
0903 08:07:50.133014 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 08:07:50.133178 (+ 164us) server_negotiation.cc:408] Connection header received
0903 08:07:50.133333 (+ 155us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:07:50.133336 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:07:50.133379 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:07:50.133448 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:07:50.134476 (+ 1028us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:50.134941 (+ 465us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:07:50.135849 (+ 908us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:50.135982 (+ 133us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:50.138360 (+ 2378us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:07:50.138378 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:07:50.138380 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:07:50.138402 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:07:50.139823 (+ 1421us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:07:50.140367 (+ 544us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:07:50.140369 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:07:50.140371 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:07:50.140408 (+ 37us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:07:50.140664 (+ 256us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:07:50.140667 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:07:50.140668 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:07:50.140795 (+ 127us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:07:50.140863 (+ 68us) server_negotiation.cc:1036] Waiting for connection context
0903 08:07:50.141188 (+ 325us) server_negotiation.cc:300] Negotiation successful
0903 08:07:50.141383 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":90,"server-negotiator.queue_time_us":48}
I20250903 08:07:50.141577 14782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:50.132767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:46669 (local address 127.0.0.1:52340)
0903 08:07:50.133077 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0903 08:07:50.133094 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0903 08:07:50.133223 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:07:50.133505 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:07:50.133510 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:07:50.134075 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:07:50.134362 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:50.134372 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:50.135089 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:07:50.135094 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:07:50.135706 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:50.135719 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:50.135882 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:50.136518 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:07:50.136535 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:07:50.138223 (+ 1688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:07:50.139962 (+ 1739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:07:50.139971 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:07:50.139977 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:07:50.140267 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:07:50.140498 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:07:50.140502 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:07:50.140504 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:07:50.140567 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:07:50.140898 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:07:50.140905 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:07:50.141027 (+ 122us) client_negotiation.cc:769] Sending connection context
0903 08:07:50.141162 (+ 135us) client_negotiation.cc:240] Negotiation successful
0903 08:07:50.141322 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":99,"threads_started":1}
I20250903 08:07:50.145498 14230 catalog_manager.cc:2232] Servicing CreateTable request from {username='kudu', principal='kudu/127.12.201.254@KRBTEST.COM'} at 127.0.0.1:52340:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20250903 08:07:50.146339 14230 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 08:07:50.151288 14370 tablet_service.cc:1468] Processing CreateTablet for tablet 153686c86c9d4ee6af8d99c6848188c6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d825552d3acf4ba29cd00137a5465f21]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20250903 08:07:50.151654 14370 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 153686c86c9d4ee6af8d99c6848188c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:07:50.151652 14654 tablet_service.cc:1468] Processing CreateTablet for tablet 153686c86c9d4ee6af8d99c6848188c6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d825552d3acf4ba29cd00137a5465f21]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20250903 08:07:50.152002 14654 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 153686c86c9d4ee6af8d99c6848188c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:07:50.154481 14745 tablet_bootstrap.cc:492] T 153686c86c9d4ee6af8d99c6848188c6 P dc9b942bcd6349a6adb9faaff24b94b3: Bootstrap starting.
I20250903 08:07:50.154495 14744 tablet_bootstrap.cc:492] T 153686c86c9d4ee6af8d99c6848188c6 P 34e13456e0c24db6a9b9b04a81897d08: Bootstrap starting.
I20250903 08:07:50.155405 14744 tablet_bootstrap.cc:654] T 153686c86c9d4ee6af8d99c6848188c6 P 34e13456e0c24db6a9b9b04a81897d08: Neither blocks nor log segments found. Creating new log.
I20250903 08:07:50.155642 14745 tablet_bootstrap.cc:654] T 153686c86c9d4ee6af8d99c6848188c6 P dc9b942bcd6349a6adb9faaff24b94b3: Neither blocks nor log segments found. Creating new log.
I20250903 08:07:50.156294 14512 tablet_service.cc:1468] Processing CreateTablet for tablet 153686c86c9d4ee6af8d99c6848188c6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d825552d3acf4ba29cd00137a5465f21]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20250903 08:07:50.156633 14512 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 153686c86c9d4ee6af8d99c6848188c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:07:50.157094 14745 tablet_bootstrap.cc:492] T 153686c86c9d4ee6af8d99c6848188c6 P dc9b942bcd6349a6adb9faaff24b94b3: No bootstrap required, opened a new log
I20250903 08:07:50.157186 14745 ts_tablet_manager.cc:1397] T 153686c86c9d4ee6af8d99c6848188c6 P dc9b942bcd6349a6adb9faaff24b94b3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250903 08:07:50.157702 14745 raft_consensus.cc:357] T 153686c86c9d4ee6af8d99c6848188c6 P dc9b942bcd6349a6adb9faaff24b94b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9b942bcd6349a6adb9faaff24b94b3" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 41925 } } peers { permanent_uuid: "b9be75cb719d4a1181d0d27104544cdc" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 35997 } } peers { permanent_uuid: "34e13456e0c24db6a9b9b04a81897d08" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 35365 } }
I20250903 08:07:50.157830 14745 raft_consensus.cc:383] T 153686c86c9d4ee6af8d99c6848188c6 P dc9b942bcd6349a6adb9faaff24b94b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:07:50.157876 14745 raft_consensus.cc:738] T 153686c86c9d4ee6af8d99c6848188c6 P dc9b942bcd6349a6adb9faaff24b94b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc9b942bcd6349a6adb9faaff24b94b3, State: Initialized, Role: FOLLOWER
I20250903 08:07:50.158008 14745 consensus_queue.cc:260] T 153686c86c9d4ee6af8d99c6848188c6 P dc9b942bcd6349a6adb9faaff24b94b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9b942bcd6349a6adb9faaff24b94b3" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 41925 } } peers { permanent_uuid: "b9be75cb719d4a1181d0d27104544cdc" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 35997 } } peers { permanent_uuid: "34e13456e0c24db6a9b9b04a81897d08" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 35365 } }
I20250903 08:07:50.158288 14745 ts_tablet_manager.cc:1428] T 153686c86c9d4ee6af8d99c6848188c6 P dc9b942bcd6349a6adb9faaff24b94b3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250903 08:07:50.158880 14743 tablet_bootstrap.cc:492] T 153686c86c9d4ee6af8d99c6848188c6 P b9be75cb719d4a1181d0d27104544cdc: Bootstrap starting.
I20250903 08:07:50.158972 14749 tablet_replica.cc:440] T 153686c86c9d4ee6af8d99c6848188c6 P dc9b942bcd6349a6adb9faaff24b94b3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9b942bcd6349a6adb9faaff24b94b3" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 41925 } } peers { permanent_uuid: "b9be75cb719d4a1181d0d27104544cdc" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 35997 } } peers { permanent_uuid: "34e13456e0c24db6a9b9b04a81897d08" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 35365 } } }
I20250903 08:07:50.159127 14749 tablet_replica.cc:443] T 153686c86c9d4ee6af8d99c6848188c6 P dc9b942bcd6349a6adb9faaff24b94b3: This TxnStatusTablet replica's current role is: FOLLOWER
I20250903 08:07:50.159824 14744 tablet_bootstrap.cc:492] T 153686c86c9d4ee6af8d99c6848188c6 P 34e13456e0c24db6a9b9b04a81897d08: No bootstrap required, opened a new log
I20250903 08:07:50.159909 14744 ts_tablet_manager.cc:1397] T 153686c86c9d4ee6af8d99c6848188c6 P 34e13456e0c24db6a9b9b04a81897d08: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20250903 08:07:50.159953 14743 tablet_bootstrap.cc:654] T 153686c86c9d4ee6af8d99c6848188c6 P b9be75cb719d4a1181d0d27104544cdc: Neither blocks nor log segments found. Creating new log.
I20250903 08:07:50.160415 14744 raft_consensus.cc:357] T 153686c86c9d4ee6af8d99c6848188c6 P 34e13456e0c24db6a9b9b04a81897d08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9b942bcd6349a6adb9faaff24b94b3" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 41925 } } peers { permanent_uuid: "b9be75cb719d4a1181d0d27104544cdc" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 35997 } } peers { permanent_uuid: "34e13456e0c24db6a9b9b04a81897d08" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 35365 } }
I20250903 08:07:50.160532 14744 raft_consensus.cc:383] T 153686c86c9d4ee6af8d99c6848188c6 P 34e13456e0c24db6a9b9b04a81897d08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:07:50.160567 14744 raft_consensus.cc:738] T 153686c86c9d4ee6af8d99c6848188c6 P 34e13456e0c24db6a9b9b04a81897d08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34e13456e0c24db6a9b9b04a81897d08, State: Initialized, Role: FOLLOWER
I20250903 08:07:50.160686 14744 consensus_queue.cc:260] T 153686c86c9d4ee6af8d99c6848188c6 P 34e13456e0c24db6a9b9b04a81897d08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9b942bcd6349a6adb9faaff24b94b3" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 41925 } } peers { permanent_uuid: "b9be75cb719d4a1181d0d27104544cdc" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 35997 } } peers { permanent_uuid: "34e13456e0c24db6a9b9b04a81897d08" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 35365 } }
I20250903 08:07:50.160959 14744 ts_tablet_manager.cc:1428] T 153686c86c9d4ee6af8d99c6848188c6 P 34e13456e0c24db6a9b9b04a81897d08: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250903 08:07:50.161178 14750 tablet_replica.cc:440] T 153686c86c9d4ee6af8d99c6848188c6 P 34e13456e0c24db6a9b9b04a81897d08: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9b942bcd6349a6adb9faaff24b94b3" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 41925 } } peers { permanent_uuid: "b9be75cb719d4a1181d0d27104544cdc" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 35997 } } peers { permanent_uuid: "34e13456e0c24db6a9b9b04a81897d08" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 35365 } } }
I20250903 08:07:50.161281 14743 tablet_bootstrap.cc:492] T 153686c86c9d4ee6af8d99c6848188c6 P b9be75cb719d4a1181d0d27104544cdc: No bootstrap required, opened a new log
I20250903 08:07:50.161288 14750 tablet_replica.cc:443] T 153686c86c9d4ee6af8d99c6848188c6 P 34e13456e0c24db6a9b9b04a81897d08: This TxnStatusTablet replica's current role is: FOLLOWER
I20250903 08:07:50.161351 14743 ts_tablet_manager.cc:1397] T 153686c86c9d4ee6af8d99c6848188c6 P b9be75cb719d4a1181d0d27104544cdc: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250903 08:07:50.161815 14743 raft_consensus.cc:357] T 153686c86c9d4ee6af8d99c6848188c6 P b9be75cb719d4a1181d0d27104544cdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9b942bcd6349a6adb9faaff24b94b3" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 41925 } } peers { permanent_uuid: "b9be75cb719d4a1181d0d27104544cdc" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 35997 } } peers { permanent_uuid: "34e13456e0c24db6a9b9b04a81897d08" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 35365 } }
I20250903 08:07:50.161906 14743 raft_consensus.cc:383] T 153686c86c9d4ee6af8d99c6848188c6 P b9be75cb719d4a1181d0d27104544cdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:07:50.161937 14743 raft_consensus.cc:738] T 153686c86c9d4ee6af8d99c6848188c6 P b9be75cb719d4a1181d0d27104544cdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9be75cb719d4a1181d0d27104544cdc, State: Initialized, Role: FOLLOWER
I20250903 08:07:50.162070 14743 consensus_queue.cc:260] T 153686c86c9d4ee6af8d99c6848188c6 P b9be75cb719d4a1181d0d27104544cdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9b942bcd6349a6adb9faaff24b94b3" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 41925 } } peers { permanent_uuid: "b9be75cb719d4a1181d0d27104544cdc" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 35997 } } peers { permanent_uuid: "34e13456e0c24db6a9b9b04a81897d08" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 35365 } }
I20250903 08:07:50.162319 14743 ts_tablet_manager.cc:1428] T 153686c86c9d4ee6af8d99c6848188c6 P b9be75cb719d4a1181d0d27104544cdc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250903 08:07:50.162931 14751 tablet_replica.cc:440] T 153686c86c9d4ee6af8d99c6848188c6 P b9be75cb719d4a1181d0d27104544cdc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9b942bcd6349a6adb9faaff24b94b3" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 41925 } } peers { permanent_uuid: "b9be75cb719d4a1181d0d27104544cdc" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 35997 } } peers { permanent_uuid: "34e13456e0c24db6a9b9b04a81897d08" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 35365 } } }
I20250903 08:07:50.163069 14751 tablet_replica.cc:443] T 153686c86c9d4ee6af8d99c6848188c6 P b9be75cb719d4a1181d0d27104544cdc: This TxnStatusTablet replica's current role is: FOLLOWER
W20250903 08:07:50.260828 14579 tablet.cc:2378] T 153686c86c9d4ee6af8d99c6848188c6 P b9be75cb719d4a1181d0d27104544cdc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250903 08:07:50.496001 14751 raft_consensus.cc:491] T 153686c86c9d4ee6af8d99c6848188c6 P b9be75cb719d4a1181d0d27104544cdc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 08:07:50.496136 14751 raft_consensus.cc:513] T 153686c86c9d4ee6af8d99c6848188c6 P b9be75cb719d4a1181d0d27104544cdc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9b942bcd6349a6adb9faaff24b94b3" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 41925 } } peers { permanent_uuid: "b9be75cb719d4a1181d0d27104544cdc" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 35997 } } peers { permanent_uuid: "34e13456e0c24db6a9b9b04a81897d08" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 35365 } }
I20250903 08:07:50.496565 14751 leader_election.cc:290] T 153686c86c9d4ee6af8d99c6848188c6 P b9be75cb719d4a1181d0d27104544cdc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc9b942bcd6349a6adb9faaff24b94b3 (127.12.201.193:41925), 34e13456e0c24db6a9b9b04a81897d08 (127.12.201.195:35365)
I20250903 08:07:50.497073 14674 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "153686c86c9d4ee6af8d99c6848188c6" candidate_uuid: "b9be75cb719d4a1181d0d27104544cdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34e13456e0c24db6a9b9b04a81897d08" is_pre_election: true
I20250903 08:07:50.497073 14390 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "153686c86c9d4ee6af8d99c6848188c6" candidate_uuid: "b9be75cb719d4a1181d0d27104544cdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc9b942bcd6349a6adb9faaff24b94b3" is_pre_election: true
I20250903 08:07:50.497231 14390 raft_consensus.cc:2466] T 153686c86c9d4ee6af8d99c6848188c6 P dc9b942bcd6349a6adb9faaff24b94b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9be75cb719d4a1181d0d27104544cdc in term 0.
I20250903 08:07:50.497231 14674 raft_consensus.cc:2466] T 153686c86c9d4ee6af8d99c6848188c6 P 34e13456e0c24db6a9b9b04a81897d08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9be75cb719d4a1181d0d27104544cdc in term 0.
I20250903 08:07:50.497565 14464 leader_election.cc:304] T 153686c86c9d4ee6af8d99c6848188c6 P b9be75cb719d4a1181d0d27104544cdc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34e13456e0c24db6a9b9b04a81897d08, b9be75cb719d4a1181d0d27104544cdc; no voters:
I20250903 08:07:50.497745 14751 raft_consensus.cc:2802] T 153686c86c9d4ee6af8d99c6848188c6 P b9be75cb719d4a1181d0d27104544cdc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 08:07:50.497807 14751 raft_consensus.cc:491] T 153686c86c9d4ee6af8d99c6848188c6 P b9be75cb719d4a1181d0d27104544cdc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 08:07:50.497836 14751 raft_consensus.cc:3058] T 153686c86c9d4ee6af8d99c6848188c6 P b9be75cb719d4a1181d0d27104544cdc [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:07:50.512265 14751 raft_consensus.cc:513] T 153686c86c9d4ee6af8d99c6848188c6 P b9be75cb719d4a1181d0d27104544cdc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9b942bcd6349a6adb9faaff24b94b3" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 41925 } } peers { permanent_uuid: "b9be75cb719d4a1181d0d27104544cdc" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 35997 } } peers { permanent_uuid: "34e13456e0c24db6a9b9b04a81897d08" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 35365 } }
I20250903 08:07:50.512818 14751 leader_election.cc:290] T 153686c86c9d4ee6af8d99c6848188c6 P b9be75cb719d4a1181d0d27104544cdc [CANDIDATE]: Term 1 election: Requested vote from peers dc9b942bcd6349a6adb9faaff24b94b3 (127.12.201.193:41925), 34e13456e0c24db6a9b9b04a81897d08 (127.12.201.195:35365)
I20250903 08:07:50.513330 14390 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "153686c86c9d4ee6af8d99c6848188c6" candidate_uuid: "b9be75cb719d4a1181d0d27104544cdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc9b942bcd6349a6adb9faaff24b94b3"
I20250903 08:07:50.513387 14674 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "153686c86c9d4ee6af8d99c6848188c6" candidate_uuid: "b9be75cb719d4a1181d0d27104544cdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34e13456e0c24db6a9b9b04a81897d08"
I20250903 08:07:50.513466 14390 raft_consensus.cc:3058] T 153686c86c9d4ee6af8d99c6848188c6 P dc9b942bcd6349a6adb9faaff24b94b3 [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:07:50.513530 14674 raft_consensus.cc:3058] T 153686c86c9d4ee6af8d99c6848188c6 P 34e13456e0c24db6a9b9b04a81897d08 [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:07:50.514533 14390 raft_consensus.cc:2466] T 153686c86c9d4ee6af8d99c6848188c6 P dc9b942bcd6349a6adb9faaff24b94b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9be75cb719d4a1181d0d27104544cdc in term 1.
I20250903 08:07:50.514858 14674 raft_consensus.cc:2466] T 153686c86c9d4ee6af8d99c6848188c6 P 34e13456e0c24db6a9b9b04a81897d08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9be75cb719d4a1181d0d27104544cdc in term 1.
I20250903 08:07:50.514968 14465 leader_election.cc:304] T 153686c86c9d4ee6af8d99c6848188c6 P b9be75cb719d4a1181d0d27104544cdc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9be75cb719d4a1181d0d27104544cdc, dc9b942bcd6349a6adb9faaff24b94b3; no voters:
I20250903 08:07:50.515268 14751 raft_consensus.cc:2802] T 153686c86c9d4ee6af8d99c6848188c6 P b9be75cb719d4a1181d0d27104544cdc [term 1 FOLLOWER]: Leader election won for term 1
I20250903 08:07:50.515439 14751 raft_consensus.cc:695] T 153686c86c9d4ee6af8d99c6848188c6 P b9be75cb719d4a1181d0d27104544cdc [term 1 LEADER]: Becoming Leader. State: Replica: b9be75cb719d4a1181d0d27104544cdc, State: Running, Role: LEADER
I20250903 08:07:50.515668 14751 consensus_queue.cc:237] T 153686c86c9d4ee6af8d99c6848188c6 P b9be75cb719d4a1181d0d27104544cdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9b942bcd6349a6adb9faaff24b94b3" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 41925 } } peers { permanent_uuid: "b9be75cb719d4a1181d0d27104544cdc" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 35997 } } peers { permanent_uuid: "34e13456e0c24db6a9b9b04a81897d08" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 35365 } }
I20250903 08:07:50.517380 14786 tablet_replica.cc:440] T 153686c86c9d4ee6af8d99c6848188c6 P b9be75cb719d4a1181d0d27104544cdc: TxnStatusTablet state changed. Reason: New leader b9be75cb719d4a1181d0d27104544cdc. Latest consensus state: current_term: 1 leader_uuid: "b9be75cb719d4a1181d0d27104544cdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9b942bcd6349a6adb9faaff24b94b3" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 41925 } } peers { permanent_uuid: "b9be75cb719d4a1181d0d27104544cdc" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 35997 } } peers { permanent_uuid: "34e13456e0c24db6a9b9b04a81897d08" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 35365 } } }
I20250903 08:07:50.517479 14786 tablet_replica.cc:443] T 153686c86c9d4ee6af8d99c6848188c6 P b9be75cb719d4a1181d0d27104544cdc: This TxnStatusTablet replica's current role is: LEADER
I20250903 08:07:50.517690 14788 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20250903 08:07:50.517822 14788 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20250903 08:07:50.518699 14230 catalog_manager.cc:5582] T 153686c86c9d4ee6af8d99c6848188c6 P b9be75cb719d4a1181d0d27104544cdc reported cstate change: term changed from 0 to 1, leader changed from <none> to b9be75cb719d4a1181d0d27104544cdc (127.12.201.194). New cstate: current_term: 1 leader_uuid: "b9be75cb719d4a1181d0d27104544cdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9b942bcd6349a6adb9faaff24b94b3" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 41925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9be75cb719d4a1181d0d27104544cdc" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 35997 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34e13456e0c24db6a9b9b04a81897d08" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 35365 } health_report { overall_health: UNKNOWN } } }
I20250903 08:07:50.532773 14782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:50.529244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.194:35997 (local address 127.0.0.1:34322)
0903 08:07:50.529398 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0903 08:07:50.529416 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0903 08:07:50.529521 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:07:50.529821 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:07:50.529823 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:07:50.529829 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0903 08:07:50.530007 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:50.530012 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:50.531021 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:07:50.531028 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:07:50.531451 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:50.531459 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:50.531580 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:50.531622 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0903 08:07:50.532515 (+ 893us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0903 08:07:50.532530 (+ 15us) client_negotiation.cc:769] Sending connection context
0903 08:07:50.532597 (+ 67us) client_negotiation.cc:240] Negotiation successful
0903 08:07:50.532662 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20250903 08:07:50.532838 14742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:50.529365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34322 (local address 127.12.201.194:35997)
0903 08:07:50.529495 (+ 130us) server_negotiation.cc:207] Beginning negotiation
0903 08:07:50.529500 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 08:07:50.529516 (+ 16us) server_negotiation.cc:408] Connection header received
0903 08:07:50.529642 (+ 126us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:07:50.529645 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:07:50.529711 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:07:50.529816 (+ 105us) server_negotiation.cc:227] Negotiated authn=TOKEN
0903 08:07:50.530109 (+ 293us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:50.530852 (+ 743us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:07:50.531564 (+ 712us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:50.531757 (+ 193us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:50.531817 (+ 60us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0903 08:07:50.532386 (+ 569us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0903 08:07:50.532507 (+ 121us) server_negotiation.cc:1036] Waiting for connection context
0903 08:07:50.532624 (+ 117us) server_negotiation.cc:300] Negotiation successful
0903 08:07:50.532712 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20250903 08:07:50.535645 14674 raft_consensus.cc:1273] T 153686c86c9d4ee6af8d99c6848188c6 P 34e13456e0c24db6a9b9b04a81897d08 [term 1 FOLLOWER]: Refusing update from remote peer b9be75cb719d4a1181d0d27104544cdc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 08:07:50.535847 14390 raft_consensus.cc:1273] T 153686c86c9d4ee6af8d99c6848188c6 P dc9b942bcd6349a6adb9faaff24b94b3 [term 1 FOLLOWER]: Refusing update from remote peer b9be75cb719d4a1181d0d27104544cdc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 08:07:50.536371 14786 consensus_queue.cc:1035] T 153686c86c9d4ee6af8d99c6848188c6 P b9be75cb719d4a1181d0d27104544cdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9b942bcd6349a6adb9faaff24b94b3" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 41925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 08:07:50.536530 14751 consensus_queue.cc:1035] T 153686c86c9d4ee6af8d99c6848188c6 P b9be75cb719d4a1181d0d27104544cdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "34e13456e0c24db6a9b9b04a81897d08" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 35365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 08:07:50.537940 14750 tablet_replica.cc:440] T 153686c86c9d4ee6af8d99c6848188c6 P 34e13456e0c24db6a9b9b04a81897d08: TxnStatusTablet state changed. Reason: New leader b9be75cb719d4a1181d0d27104544cdc. Latest consensus state: current_term: 1 leader_uuid: "b9be75cb719d4a1181d0d27104544cdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9b942bcd6349a6adb9faaff24b94b3" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 41925 } } peers { permanent_uuid: "b9be75cb719d4a1181d0d27104544cdc" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 35997 } } peers { permanent_uuid: "34e13456e0c24db6a9b9b04a81897d08" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 35365 } } }
I20250903 08:07:50.538048 14750 tablet_replica.cc:443] T 153686c86c9d4ee6af8d99c6848188c6 P 34e13456e0c24db6a9b9b04a81897d08: This TxnStatusTablet replica's current role is: FOLLOWER
I20250903 08:07:50.539744 14749 tablet_replica.cc:440] T 153686c86c9d4ee6af8d99c6848188c6 P dc9b942bcd6349a6adb9faaff24b94b3: TxnStatusTablet state changed. Reason: New leader b9be75cb719d4a1181d0d27104544cdc. Latest consensus state: current_term: 1 leader_uuid: "b9be75cb719d4a1181d0d27104544cdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9b942bcd6349a6adb9faaff24b94b3" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 41925 } } peers { permanent_uuid: "b9be75cb719d4a1181d0d27104544cdc" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 35997 } } peers { permanent_uuid: "34e13456e0c24db6a9b9b04a81897d08" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 35365 } } }
I20250903 08:07:50.539858 14749 tablet_replica.cc:443] T 153686c86c9d4ee6af8d99c6848188c6 P dc9b942bcd6349a6adb9faaff24b94b3: This TxnStatusTablet replica's current role is: FOLLOWER
I20250903 08:07:50.540920 14786 tablet_replica.cc:440] T 153686c86c9d4ee6af8d99c6848188c6 P b9be75cb719d4a1181d0d27104544cdc: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b9be75cb719d4a1181d0d27104544cdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9b942bcd6349a6adb9faaff24b94b3" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 41925 } } peers { permanent_uuid: "b9be75cb719d4a1181d0d27104544cdc" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 35997 } } peers { permanent_uuid: "34e13456e0c24db6a9b9b04a81897d08" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 35365 } } }
I20250903 08:07:50.541013 14786 tablet_replica.cc:443] T 153686c86c9d4ee6af8d99c6848188c6 P b9be75cb719d4a1181d0d27104544cdc: This TxnStatusTablet replica's current role is: LEADER
I20250903 08:07:50.541122 14786 tablet_replica.cc:440] T 153686c86c9d4ee6af8d99c6848188c6 P b9be75cb719d4a1181d0d27104544cdc: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b9be75cb719d4a1181d0d27104544cdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9b942bcd6349a6adb9faaff24b94b3" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 41925 } } peers { permanent_uuid: "b9be75cb719d4a1181d0d27104544cdc" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 35997 } } peers { permanent_uuid: "34e13456e0c24db6a9b9b04a81897d08" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 35365 } } }
I20250903 08:07:50.541186 14786 tablet_replica.cc:443] T 153686c86c9d4ee6af8d99c6848188c6 P b9be75cb719d4a1181d0d27104544cdc: This TxnStatusTablet replica's current role is: LEADER
I20250903 08:07:50.543473 14775 tablet_replica.cc:440] T 153686c86c9d4ee6af8d99c6848188c6 P 34e13456e0c24db6a9b9b04a81897d08: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b9be75cb719d4a1181d0d27104544cdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9b942bcd6349a6adb9faaff24b94b3" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 41925 } } peers { permanent_uuid: "b9be75cb719d4a1181d0d27104544cdc" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 35997 } } peers { permanent_uuid: "34e13456e0c24db6a9b9b04a81897d08" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 35365 } } }
I20250903 08:07:50.543583 14775 tablet_replica.cc:443] T 153686c86c9d4ee6af8d99c6848188c6 P 34e13456e0c24db6a9b9b04a81897d08: This TxnStatusTablet replica's current role is: FOLLOWER
I20250903 08:07:50.545004 14775 consensus_queue.cc:1035] T 4ab1cbf6753a46aeb78c5783e0e2da6c P 34e13456e0c24db6a9b9b04a81897d08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9be75cb719d4a1181d0d27104544cdc" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 35997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 08:07:50.546567 14749 tablet_replica.cc:440] T 153686c86c9d4ee6af8d99c6848188c6 P dc9b942bcd6349a6adb9faaff24b94b3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b9be75cb719d4a1181d0d27104544cdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9b942bcd6349a6adb9faaff24b94b3" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 41925 } } peers { permanent_uuid: "b9be75cb719d4a1181d0d27104544cdc" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 35997 } } peers { permanent_uuid: "34e13456e0c24db6a9b9b04a81897d08" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 35365 } } }
I20250903 08:07:50.546679 14749 tablet_replica.cc:443] T 153686c86c9d4ee6af8d99c6848188c6 P dc9b942bcd6349a6adb9faaff24b94b3: This TxnStatusTablet replica's current role is: FOLLOWER
I20250903 08:07:50.549104 14775 consensus_queue.cc:1035] T 4ab1cbf6753a46aeb78c5783e0e2da6c P 34e13456e0c24db6a9b9b04a81897d08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9b942bcd6349a6adb9faaff24b94b3" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 41925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 08:07:50.559607 14805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:50.555674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45308 (local address 127.12.201.195:35365)
0903 08:07:50.556131 (+ 457us) server_negotiation.cc:207] Beginning negotiation
0903 08:07:50.556135 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:07:50.556144 (+ 9us) server_negotiation.cc:408] Connection header received
0903 08:07:50.556206 (+ 62us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:07:50.556209 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:07:50.556257 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:07:50.556344 (+ 87us) server_negotiation.cc:227] Negotiated authn=TOKEN
0903 08:07:50.556822 (+ 478us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:50.557313 (+ 491us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:07:50.558505 (+ 1192us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:50.558648 (+ 143us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:50.558697 (+ 49us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0903 08:07:50.559192 (+ 495us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0903 08:07:50.559279 (+ 87us) server_negotiation.cc:1036] Waiting for connection context
0903 08:07:50.559373 (+ 94us) server_negotiation.cc:300] Negotiation successful
0903 08:07:50.559421 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":108,"threads_started":1}
I20250903 08:07:50.569448 14742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:50.566618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.195:59031 (local address 127.12.201.194:35997)
0903 08:07:50.566740 (+ 122us) server_negotiation.cc:207] Beginning negotiation
0903 08:07:50.566745 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 08:07:50.566957 (+ 212us) server_negotiation.cc:408] Connection header received
0903 08:07:50.567104 (+ 147us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:07:50.567107 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:07:50.567153 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:07:50.567223 (+ 70us) server_negotiation.cc:227] Negotiated authn=TOKEN
0903 08:07:50.567623 (+ 400us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:50.568058 (+ 435us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:07:50.568914 (+ 856us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:50.569045 (+ 131us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:50.569079 (+ 34us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0903 08:07:50.569139 (+ 60us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0903 08:07:50.569195 (+ 56us) server_negotiation.cc:1036] Waiting for connection context
0903 08:07:50.569303 (+ 108us) server_negotiation.cc:300] Negotiation successful
0903 08:07:50.569342 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20250903 08:07:50.569526 14808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:50.566501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.194:35997 (local address 127.12.201.195:59031)
0903 08:07:50.566871 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0903 08:07:50.566885 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0903 08:07:50.567013 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:07:50.567240 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:07:50.567244 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:07:50.567252 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0903 08:07:50.567532 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:50.567539 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:50.568161 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:07:50.568165 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:07:50.568813 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:50.568822 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:50.568943 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:50.568982 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0903 08:07:50.569221 (+ 239us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0903 08:07:50.569227 (+ 6us) client_negotiation.cc:769] Sending connection context
0903 08:07:50.569292 (+ 65us) client_negotiation.cc:240] Negotiation successful
0903 08:07:50.569366 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":89,"threads_started":1}
I20250903 08:07:50.594609 14805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:50.591465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.194:35197 (local address 127.12.201.195:35365)
0903 08:07:50.591595 (+ 130us) server_negotiation.cc:207] Beginning negotiation
0903 08:07:50.591601 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0903 08:07:50.591824 (+ 223us) server_negotiation.cc:408] Connection header received
0903 08:07:50.591952 (+ 128us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:07:50.591955 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:07:50.592000 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:07:50.592067 (+ 67us) server_negotiation.cc:227] Negotiated authn=TOKEN
0903 08:07:50.592565 (+ 498us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:50.593142 (+ 577us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:07:50.594050 (+ 908us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:50.594187 (+ 137us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:50.594237 (+ 50us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0903 08:07:50.594301 (+ 64us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0903 08:07:50.594361 (+ 60us) server_negotiation.cc:1036] Waiting for connection context
0903 08:07:50.594470 (+ 109us) server_negotiation.cc:300] Negotiation successful
0903 08:07:50.594508 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20250903 08:07:50.595811 14812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:50.591406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.195:35365 (local address 127.12.201.194:35197)
0903 08:07:50.591744 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0903 08:07:50.591757 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0903 08:07:50.591858 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:07:50.592098 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:07:50.592101 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:07:50.592111 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0903 08:07:50.592460 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:50.592468 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:50.593260 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:07:50.593264 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:07:50.593935 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:50.593944 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:50.594065 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:50.594108 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0903 08:07:50.594382 (+ 274us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0903 08:07:50.594387 (+ 5us) client_negotiation.cc:769] Sending connection context
0903 08:07:50.594462 (+ 75us) client_negotiation.cc:240] Negotiation successful
0903 08:07:50.594524 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":93,"threads_started":1}
I20250903 08:07:50.619184 14231 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52336:
table { table_name: "test-table" } modify_external_catalogs: true
I20250903 08:07:50.619387 14231 catalog_manager.cc:2730] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52336:
table { table_name: "test-table" } modify_external_catalogs: true
I20250903 08:07:50.621670 14231 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 5b996e3f77d84707b4bfc52734aeb4c9: Sending DeleteTablet for 3 replicas of tablet 4ab1cbf6753a46aeb78c5783e0e2da6c
I20250903 08:07:50.622227 14370 tablet_service.cc:1515] Processing DeleteTablet for tablet 4ab1cbf6753a46aeb78c5783e0e2da6c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 08:07:50 UTC) from {username='kudu', principal='kudu/127.12.201.254@KRBTEST.COM'} at 127.0.0.1:49938
I20250903 08:07:50.622321 14512 tablet_service.cc:1515] Processing DeleteTablet for tablet 4ab1cbf6753a46aeb78c5783e0e2da6c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 08:07:50 UTC) from {username='kudu', principal='kudu/127.12.201.254@KRBTEST.COM'} at 127.0.0.1:34316
I20250903 08:07:50.622699 14815 tablet_replica.cc:331] T 4ab1cbf6753a46aeb78c5783e0e2da6c P dc9b942bcd6349a6adb9faaff24b94b3: stopping tablet replica
I20250903 08:07:50.622864 14815 raft_consensus.cc:2241] T 4ab1cbf6753a46aeb78c5783e0e2da6c P dc9b942bcd6349a6adb9faaff24b94b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 08:07:50.623049 14815 raft_consensus.cc:2270] T 4ab1cbf6753a46aeb78c5783e0e2da6c P dc9b942bcd6349a6adb9faaff24b94b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 08:07:50.623096 14654 tablet_service.cc:1515] Processing DeleteTablet for tablet 4ab1cbf6753a46aeb78c5783e0e2da6c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 08:07:50 UTC) from {username='kudu', principal='kudu/127.12.201.254@KRBTEST.COM'} at 127.0.0.1:45306
I20250903 08:07:50.623823 14815 ts_tablet_manager.cc:1905] T 4ab1cbf6753a46aeb78c5783e0e2da6c P dc9b942bcd6349a6adb9faaff24b94b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 08:07:50.624290 14817 tablet_replica.cc:331] T 4ab1cbf6753a46aeb78c5783e0e2da6c P 34e13456e0c24db6a9b9b04a81897d08: stopping tablet replica
I20250903 08:07:50.624420 14817 raft_consensus.cc:2241] T 4ab1cbf6753a46aeb78c5783e0e2da6c P 34e13456e0c24db6a9b9b04a81897d08 [term 1 LEADER]: Raft consensus shutting down.
I20250903 08:07:50.624717 14817 raft_consensus.cc:2270] T 4ab1cbf6753a46aeb78c5783e0e2da6c P 34e13456e0c24db6a9b9b04a81897d08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 08:07:50.626253 14817 ts_tablet_manager.cc:1905] T 4ab1cbf6753a46aeb78c5783e0e2da6c P 34e13456e0c24db6a9b9b04a81897d08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 08:07:50.627173 14816 tablet_replica.cc:331] T 4ab1cbf6753a46aeb78c5783e0e2da6c P b9be75cb719d4a1181d0d27104544cdc: stopping tablet replica
I20250903 08:07:50.627297 14816 raft_consensus.cc:2241] T 4ab1cbf6753a46aeb78c5783e0e2da6c P b9be75cb719d4a1181d0d27104544cdc [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 08:07:50.627440 14816 raft_consensus.cc:2270] T 4ab1cbf6753a46aeb78c5783e0e2da6c P b9be75cb719d4a1181d0d27104544cdc [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 08:07:50.628151 14816 ts_tablet_manager.cc:1905] T 4ab1cbf6753a46aeb78c5783e0e2da6c P b9be75cb719d4a1181d0d27104544cdc: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 03 08:07:50 dist-test-slave-27r9 krb5kdc[14183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886869, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.201.193@KRBTEST.COM
I20250903 08:07:50.638703 14819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:50.625137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49958 (local address 127.12.201.193:41925)
0903 08:07:50.628526 (+ 3389us) server_negotiation.cc:207] Beginning negotiation
0903 08:07:50.628531 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 08:07:50.628890 (+ 359us) server_negotiation.cc:408] Connection header received
0903 08:07:50.628934 (+ 44us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:07:50.628938 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:07:50.628986 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:07:50.629061 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:07:50.630186 (+ 1125us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:50.631012 (+ 826us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:07:50.631860 (+ 848us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:50.632446 (+ 586us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:50.634765 (+ 2319us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:07:50.634792 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:07:50.634806 (+ 14us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:07:50.634837 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:07:50.637022 (+ 2185us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:07:50.637493 (+ 471us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:07:50.637500 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:07:50.637503 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:07:50.637570 (+ 67us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:07:50.637793 (+ 223us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:07:50.637796 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:07:50.637798 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:07:50.638018 (+ 220us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:07:50.638226 (+ 208us) server_negotiation.cc:1036] Waiting for connection context
0903 08:07:50.638378 (+ 152us) server_negotiation.cc:300] Negotiation successful
0903 08:07:50.638516 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3300,"thread_start_us":57,"threads_started":1}
W20250903 08:07:50.639281 14421 server_base.cc:1129] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49958
I20250903 08:07:50.649078 14280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:50.640973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52350 (local address 127.12.201.254:46669)
0903 08:07:50.641102 (+ 129us) server_negotiation.cc:207] Beginning negotiation
0903 08:07:50.641106 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:07:50.641514 (+ 408us) server_negotiation.cc:408] Connection header received
0903 08:07:50.641612 (+ 98us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:07:50.641616 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:07:50.641672 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:07:50.641757 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:07:50.642656 (+ 899us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:50.643496 (+ 840us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:07:50.644313 (+ 817us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:50.644878 (+ 565us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:50.645737 (+ 859us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:07:50.645758 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:07:50.645761 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:07:50.645787 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:07:50.647568 (+ 1781us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:07:50.647997 (+ 429us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:07:50.648001 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:07:50.648002 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:07:50.648049 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:07:50.648288 (+ 239us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:07:50.648291 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:07:50.648293 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:07:50.648476 (+ 183us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:07:50.648608 (+ 132us) server_negotiation.cc:1036] Waiting for connection context
0903 08:07:50.648793 (+ 185us) server_negotiation.cc:300] Negotiation successful
0903 08:07:50.648909 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20250903 08:07:50.649462 14231 server_base.cc:1129] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52350
I20250903 08:07:50.650694 13095 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUML90r/build/debug/bin/kudu with pid 14305
W20250903 08:07:50.727851 14217 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.201.193:41925 (error 108)
I20250903 08:07:50.728907 13095 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUML90r/build/debug/bin/kudu with pid 14447
I20250903 08:07:50.731304 14824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:50.730836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.193:41925 (local address 127.0.0.1:49960)
0903 08:07:50.731107 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0903 08:07:50.731188 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.201.193:41925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":58,"threads_started":1}
W20250903 08:07:50.731536 14217 catalog_manager.cc:4671] TS dc9b942bcd6349a6adb9faaff24b94b3 (127.12.201.193:41925): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4ab1cbf6753a46aeb78c5783e0e2da6c: Network error: Client connection negotiation failed: client connection to 127.12.201.193:41925: connect: Connection refused (error 111)
I20250903 08:07:50.771221 14817 ts_tablet_manager.cc:1918] T 4ab1cbf6753a46aeb78c5783e0e2da6c P 34e13456e0c24db6a9b9b04a81897d08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20250903 08:07:50.771325 14817 log.cc:1199] T 4ab1cbf6753a46aeb78c5783e0e2da6c P 34e13456e0c24db6a9b9b04a81897d08: Deleting WAL directory at /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756886860253814-13095-0/minicluster-data/ts-2/wal/wals/4ab1cbf6753a46aeb78c5783e0e2da6c
I20250903 08:07:50.771905 14817 ts_tablet_manager.cc:1939] T 4ab1cbf6753a46aeb78c5783e0e2da6c P 34e13456e0c24db6a9b9b04a81897d08: Deleting consensus metadata
I20250903 08:07:50.784617 14824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:50.784262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.193:41925 (local address 127.0.0.1:49974)
0903 08:07:50.784460 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0903 08:07:50.784526 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.201.193:41925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20250903 08:07:50.785346 14824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:50.785131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.193:41925 (local address 127.0.0.1:49976)
0903 08:07:50.785244 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0903 08:07:50.785289 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.201.193:41925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20250903 08:07:50.790738 14216 catalog_manager.cc:4928] TS 34e13456e0c24db6a9b9b04a81897d08 (127.12.201.195:35365): tablet 4ab1cbf6753a46aeb78c5783e0e2da6c (table test-table [id=82c1319f7a594578a286a84b3f4677a4]) successfully deleted
I20250903 08:07:50.818959 13095 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUML90r/build/debug/bin/kudu with pid 14589
I20250903 08:07:50.820451 14824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:50.820039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.194:35997 (local address 127.0.0.1:34324)
0903 08:07:50.820282 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0903 08:07:50.820339 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.201.194:35997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
I20250903 08:07:50.825745 13095 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUML90r/build/debug/bin/kudu with pid 14199
2025-09-03T08:07:50Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4117 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 03 08:07:52 dist-test-slave-27r9 krb5kdc[14827](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 03 08:07:52 dist-test-slave-27r9 krb5kdc[14827](info): set up 2 sockets
Sep 03 08:07:52 dist-test-slave-27r9 krb5kdc[14827](info): commencing operation
krb5kdc: starting...
W20250903 08:07:54.795313 13095 mini_kdc.cc:121] Time spent starting KDC: real 3.946s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/krb5kdc/test-user.keytab.
Sep 03 08:07:54 dist-test-slave-27r9 krb5kdc[14827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-09-03T08:07:54Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-03T08:07:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.201.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.201.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.201.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.201.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.201.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.201.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.201.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.201.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/krb5kdc/kudu.keytab.
I20250903 08:07:54.924166 13095 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUML90r/build/debug/bin/kudu
/tmp/dist-test-taskUML90r/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.201.254:36815
--webserver_interface=127.12.201.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.201.212:45945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.201.254:36815
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 08:07:55.031422 14843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:07:55.031656 14843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:07:55.031754 14843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:07:55.034885 14843 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250903 08:07:55.034947 14843 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20250903 08:07:55.034998 14843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 08:07:55.035038 14843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:07:55.035079 14843 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250903 08:07:55.035118 14843 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250903 08:07:55.039320 14843 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.201.212:45945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.201.254:36815
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.201.254:36815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.201.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14843
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 07:50:08 UTC on 5fd53c4cbb9d
build id 7874
I20250903 08:07:55.040457 14843 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:07:55.041350 14843 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:07:55.047120 14848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:07:55.047211 14843 server_base.cc:1047] running on GCE node
W20250903 08:07:55.047120 14851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:07:55.047144 14849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:07:55.047848 14843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:07:55.048781 14843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:07:55.049933 14843 hybrid_clock.cc:648] HybridClock initialized: now 1756886875049910 us; error 36 us; skew 500 ppm
Sep 03 08:07:55 dist-test-slave-27r9 krb5kdc[14827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886875, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 08:07:55.052402 14843 init.cc:377] Logged in from keytab as kudu/127.12.201.254@KRBTEST.COM (short username kudu)
I20250903 08:07:55.053416 14843 webserver.cc:480] Webserver started at http://127.12.201.254:33271/ using document root <none> and password file <none>
I20250903 08:07:55.053985 14843 fs_manager.cc:362] Metadata directory not provided
I20250903 08:07:55.054040 14843 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:07:55.054236 14843 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:07:55.056023 14843 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/minicluster-data/master-0/data/instance:
uuid: "911c1d9d6e52403ca64f3910291fda10"
format_stamp: "Formatted at 2025-09-03 08:07:55 on dist-test-slave-27r9"
server_key: "c67d55c2481a336c776d90245fd63c46"
server_key_iv: "fe1502ab3a2ef32ff18d8de4a82fcf93"
server_key_version: "encryptionkey@0"
I20250903 08:07:55.056558 14843 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/minicluster-data/master-0/wal/instance:
uuid: "911c1d9d6e52403ca64f3910291fda10"
format_stamp: "Formatted at 2025-09-03 08:07:55 on dist-test-slave-27r9"
server_key: "c67d55c2481a336c776d90245fd63c46"
server_key_iv: "fe1502ab3a2ef32ff18d8de4a82fcf93"
server_key_version: "encryptionkey@0"
I20250903 08:07:55.059968 14843 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250903 08:07:55.062251 14858 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:07:55.063318 14843 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250903 08:07:55.063429 14843 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/minicluster-data/master-0/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/minicluster-data/master-0/wal
uuid: "911c1d9d6e52403ca64f3910291fda10"
format_stamp: "Formatted at 2025-09-03 08:07:55 on dist-test-slave-27r9"
server_key: "c67d55c2481a336c776d90245fd63c46"
server_key_iv: "fe1502ab3a2ef32ff18d8de4a82fcf93"
server_key_version: "encryptionkey@0"
I20250903 08:07:55.063520 14843 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:07:55.077020 14843 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:07:55.079844 14843 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:07:55.080029 14843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:07:55.087957 14843 rpc_server.cc:307] RPC server started. Bound to: 127.12.201.254:36815
I20250903 08:07:55.087954 14920 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.201.254:36815 every 8 connection(s)
I20250903 08:07:55.088980 14843 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/minicluster-data/master-0/data/info.pb
I20250903 08:07:55.091881 14921 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:07:55.096793 14921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 911c1d9d6e52403ca64f3910291fda10: Bootstrap starting.
I20250903 08:07:55.099089 14921 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 911c1d9d6e52403ca64f3910291fda10: Neither blocks nor log segments found. Creating new log.
I20250903 08:07:55.099272 13095 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUML90r/build/debug/bin/kudu as pid 14843
I20250903 08:07:55.099381 13095 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/minicluster-data/master-0/wal/instance
I20250903 08:07:55.099588 13095 external_mini_cluster.cc:1468] Setting key ec577fe8623019465d47ba0e75fc166c
I20250903 08:07:55.099864 14921 log.cc:826] T 00000000000000000000000000000000 P 911c1d9d6e52403ca64f3910291fda10: Log is configured to *not* fsync() on all Append() calls
I20250903 08:07:55.101981 14921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 911c1d9d6e52403ca64f3910291fda10: No bootstrap required, opened a new log
I20250903 08:07:55.105005 14921 raft_consensus.cc:357] T 00000000000000000000000000000000 P 911c1d9d6e52403ca64f3910291fda10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "911c1d9d6e52403ca64f3910291fda10" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 36815 } }
I20250903 08:07:55.105225 14921 raft_consensus.cc:383] T 00000000000000000000000000000000 P 911c1d9d6e52403ca64f3910291fda10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:07:55.105332 14921 raft_consensus.cc:738] T 00000000000000000000000000000000 P 911c1d9d6e52403ca64f3910291fda10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 911c1d9d6e52403ca64f3910291fda10, State: Initialized, Role: FOLLOWER
I20250903 08:07:55.105880 14921 consensus_queue.cc:260] T 00000000000000000000000000000000 P 911c1d9d6e52403ca64f3910291fda10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "911c1d9d6e52403ca64f3910291fda10" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 36815 } }
I20250903 08:07:55.106052 14921 raft_consensus.cc:397] T 00000000000000000000000000000000 P 911c1d9d6e52403ca64f3910291fda10 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250903 08:07:55.106133 14921 raft_consensus.cc:491] T 00000000000000000000000000000000 P 911c1d9d6e52403ca64f3910291fda10 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250903 08:07:55.106249 14921 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 911c1d9d6e52403ca64f3910291fda10 [term 0 FOLLOWER]: Advancing to term 1
Sep 03 08:07:55 dist-test-slave-27r9 krb5kdc[14827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.201.254@KRBTEST.COM
I20250903 08:07:55.107152 14921 raft_consensus.cc:513] T 00000000000000000000000000000000 P 911c1d9d6e52403ca64f3910291fda10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "911c1d9d6e52403ca64f3910291fda10" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 36815 } }
I20250903 08:07:55.107489 14921 leader_election.cc:304] T 00000000000000000000000000000000 P 911c1d9d6e52403ca64f3910291fda10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 911c1d9d6e52403ca64f3910291fda10; no voters:
I20250903 08:07:55.107810 14921 leader_election.cc:290] T 00000000000000000000000000000000 P 911c1d9d6e52403ca64f3910291fda10 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250903 08:07:55.108166 14926 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 911c1d9d6e52403ca64f3910291fda10 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 08:07:55.108372 14926 raft_consensus.cc:695] T 00000000000000000000000000000000 P 911c1d9d6e52403ca64f3910291fda10 [term 1 LEADER]: Becoming Leader. State: Replica: 911c1d9d6e52403ca64f3910291fda10, State: Running, Role: LEADER
I20250903 08:07:55.108744 14926 consensus_queue.cc:237] T 00000000000000000000000000000000 P 911c1d9d6e52403ca64f3910291fda10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "911c1d9d6e52403ca64f3910291fda10" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 36815 } }
I20250903 08:07:55.109117 14921 sys_catalog.cc:564] T 00000000000000000000000000000000 P 911c1d9d6e52403ca64f3910291fda10 [sys.catalog]: configured and running, proceeding with master startup.
I20250903 08:07:55.110392 14927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 911c1d9d6e52403ca64f3910291fda10 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "911c1d9d6e52403ca64f3910291fda10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "911c1d9d6e52403ca64f3910291fda10" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 36815 } } }
I20250903 08:07:55.110522 14927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 911c1d9d6e52403ca64f3910291fda10 [sys.catalog]: This master's current role is: LEADER
I20250903 08:07:55.110392 14928 sys_catalog.cc:455] T 00000000000000000000000000000000 P 911c1d9d6e52403ca64f3910291fda10 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 911c1d9d6e52403ca64f3910291fda10. Latest consensus state: current_term: 1 leader_uuid: "911c1d9d6e52403ca64f3910291fda10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "911c1d9d6e52403ca64f3910291fda10" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 36815 } } }
I20250903 08:07:55.110738 14928 sys_catalog.cc:458] T 00000000000000000000000000000000 P 911c1d9d6e52403ca64f3910291fda10 [sys.catalog]: This master's current role is: LEADER
I20250903 08:07:55.111145 14935 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250903 08:07:55.113039 14925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:55.100629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53126 (local address 127.12.201.254:36815)
0903 08:07:55.101045 (+ 416us) server_negotiation.cc:207] Beginning negotiation
0903 08:07:55.101058 (+ 13us) server_negotiation.cc:400] Waiting for connection header
0903 08:07:55.101097 (+ 39us) server_negotiation.cc:408] Connection header received
0903 08:07:55.101756 (+ 659us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:07:55.101779 (+ 23us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:07:55.102135 (+ 356us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:07:55.102466 (+ 331us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:07:55.103205 (+ 739us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:55.104392 (+ 1187us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:07:55.105056 (+ 664us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:55.105329 (+ 273us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:55.107557 (+ 2228us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:07:55.107580 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:07:55.107597 (+ 17us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:07:55.107638 (+ 41us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:07:55.110069 (+ 2431us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:07:55.110634 (+ 565us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:07:55.110640 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:07:55.110647 (+ 7us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:07:55.110735 (+ 88us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:07:55.111074 (+ 339us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:07:55.111079 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:07:55.111081 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:07:55.111553 (+ 472us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:07:55.111753 (+ 200us) server_negotiation.cc:1036] Waiting for connection context
0903 08:07:55.112091 (+ 338us) server_negotiation.cc:300] Negotiation successful
0903 08:07:55.112383 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":125,"threads_started":1}
I20250903 08:07:55.114693 14935 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250903 08:07:55.119853 14935 catalog_manager.cc:1349] Generated new cluster ID: e65952480e864fb8b3aca645201e13f9
I20250903 08:07:55.119957 14935 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250903 08:07:55.132566 14935 catalog_manager.cc:1372] Generated new certificate authority record
I20250903 08:07:55.133435 14935 catalog_manager.cc:1506] Loading token signing keys...
I20250903 08:07:55.140666 14935 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 911c1d9d6e52403ca64f3910291fda10: Generated new TSK 0
I20250903 08:07:55.141350 14935 catalog_manager.cc:1516] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.201.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.201.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.201.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.201.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.201.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.201.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.201.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.201.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/krb5kdc/kudu.keytab.
I20250903 08:07:55.194217 13095 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUML90r/build/debug/bin/kudu
/tmp/dist-test-taskUML90r/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.201.193:0
--local_ip_for_outbound_sockets=127.12.201.193
--webserver_interface=127.12.201.193
--webserver_port=0
--tserver_master_addrs=127.12.201.254:36815
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.201.212:45945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 08:07:55.299314 14949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:07:55.299528 14949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:07:55.299592 14949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:07:55.302716 14949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 08:07:55.302773 14949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:07:55.302826 14949 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20250903 08:07:55.302842 14949 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20250903 08:07:55.302884 14949 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.201.193
I20250903 08:07:55.307125 14949 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.201.212:45945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.201.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.201.193
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.201.254:36815
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14949
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.201.193
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 07:50:08 UTC on 5fd53c4cbb9d
build id 7874
I20250903 08:07:55.308216 14949 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:07:55.309116 14949 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:07:55.315582 14954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:07:55.315582 14957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:07:55.315584 14955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:07:55.315963 14949 server_base.cc:1047] running on GCE node
I20250903 08:07:55.316315 14949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:07:55.316825 14949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:07:55.317966 14949 hybrid_clock.cc:648] HybridClock initialized: now 1756886875317955 us; error 26 us; skew 500 ppm
Sep 03 08:07:55 dist-test-slave-27r9 krb5kdc[14827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886875, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 08:07:55.320600 14949 init.cc:377] Logged in from keytab as kudu/127.12.201.193@KRBTEST.COM (short username kudu)
I20250903 08:07:55.321547 14949 webserver.cc:480] Webserver started at http://127.12.201.193:36891/ using document root <none> and password file <none>
I20250903 08:07:55.322059 14949 fs_manager.cc:362] Metadata directory not provided
I20250903 08:07:55.322108 14949 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:07:55.322294 14949 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:07:55.323979 14949 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/minicluster-data/ts-0/data/instance:
uuid: "4fe0343417e4407cacb543879bd8a7e9"
format_stamp: "Formatted at 2025-09-03 08:07:55 on dist-test-slave-27r9"
server_key: "7bd2d185f78be8dbec473c0f57ea7f84"
server_key_iv: "461011e56e1da26f7552aa906a32be24"
server_key_version: "encryptionkey@0"
I20250903 08:07:55.324437 14949 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/minicluster-data/ts-0/wal/instance:
uuid: "4fe0343417e4407cacb543879bd8a7e9"
format_stamp: "Formatted at 2025-09-03 08:07:55 on dist-test-slave-27r9"
server_key: "7bd2d185f78be8dbec473c0f57ea7f84"
server_key_iv: "461011e56e1da26f7552aa906a32be24"
server_key_version: "encryptionkey@0"
I20250903 08:07:55.327606 14949 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250903 08:07:55.329658 14964 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:07:55.330577 14949 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250903 08:07:55.330669 14949 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/minicluster-data/ts-0/wal
uuid: "4fe0343417e4407cacb543879bd8a7e9"
format_stamp: "Formatted at 2025-09-03 08:07:55 on dist-test-slave-27r9"
server_key: "7bd2d185f78be8dbec473c0f57ea7f84"
server_key_iv: "461011e56e1da26f7552aa906a32be24"
server_key_version: "encryptionkey@0"
I20250903 08:07:55.330755 14949 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:07:55.340814 14949 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:07:55.343541 14949 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:07:55.343698 14949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:07:55.344978 14949 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 08:07:55.345026 14949 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 08:07:55.345065 14949 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 08:07:55.345100 14949 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 03 08:07:55 dist-test-slave-27r9 krb5kdc[14827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886875, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.193@KRBTEST.COM for kudu/127.12.201.254@KRBTEST.COM
I20250903 08:07:55.355298 14949 rpc_server.cc:307] RPC server started. Bound to: 127.12.201.193:39419
I20250903 08:07:55.355366 15079 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.201.193:39419 every 8 connection(s)
I20250903 08:07:55.356359 14949 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/minicluster-data/ts-0/data/info.pb
I20250903 08:07:55.356945 14925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:55.345677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.193:51093 (local address 127.12.201.254:36815)
0903 08:07:55.345812 (+ 135us) server_negotiation.cc:207] Beginning negotiation
0903 08:07:55.345817 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 08:07:55.346492 (+ 675us) server_negotiation.cc:408] Connection header received
0903 08:07:55.347420 (+ 928us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:07:55.347426 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:07:55.347493 (+ 67us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:07:55.347596 (+ 103us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:07:55.349207 (+ 1611us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:55.349869 (+ 662us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:07:55.350782 (+ 913us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:55.351022 (+ 240us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:55.353108 (+ 2086us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:07:55.353131 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:07:55.353137 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:07:55.353162 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:07:55.355000 (+ 1838us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:07:55.355527 (+ 527us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:07:55.355531 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:07:55.355533 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:07:55.355582 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:07:55.355828 (+ 246us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:07:55.355831 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:07:55.355833 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:07:55.356013 (+ 180us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:07:55.356128 (+ 115us) server_negotiation.cc:1036] Waiting for connection context
0903 08:07:55.356616 (+ 488us) server_negotiation.cc:300] Negotiation successful
0903 08:07:55.356762 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20250903 08:07:55.357509 14973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:55.345919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:36815 (local address 127.12.201.193:51093)
0903 08:07:55.346336 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0903 08:07:55.346376 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0903 08:07:55.347187 (+ 811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:07:55.347820 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:07:55.347833 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:07:55.348347 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:07:55.349045 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:55.349063 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:55.349991 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:07:55.349998 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:07:55.350658 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:55.350670 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:55.350883 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:55.351528 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:07:55.351562 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:07:55.352897 (+ 1335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:07:55.355109 (+ 2212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:07:55.355114 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:07:55.355125 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:07:55.355420 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:07:55.355663 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:07:55.355665 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:07:55.355667 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:07:55.355749 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:07:55.356144 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:07:55.356150 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:07:55.356365 (+ 215us) client_negotiation.cc:769] Sending connection context
0903 08:07:55.356560 (+ 195us) client_negotiation.cc:240] Negotiation successful
0903 08:07:55.356805 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":112,"threads_started":1}
I20250903 08:07:55.358594 15080 heartbeater.cc:344] Connected to a master server at 127.12.201.254:36815
I20250903 08:07:55.358763 15080 heartbeater.cc:461] Registering TS with master...
I20250903 08:07:55.359375 15080 heartbeater.cc:507] Master 127.12.201.254:36815 requested a full tablet report, sending...
I20250903 08:07:55.359491 13095 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUML90r/build/debug/bin/kudu as pid 14949
I20250903 08:07:55.359615 13095 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/minicluster-data/ts-0/wal/instance
I20250903 08:07:55.359830 13095 external_mini_cluster.cc:1468] Setting key 51f8fbafdda1c2f1c66d16257dc055ae
I20250903 08:07:55.361312 14875 ts_manager.cc:194] Registered new tserver with Master: 4fe0343417e4407cacb543879bd8a7e9 (127.12.201.193:39419)
I20250903 08:07:55.362612 14875 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.201.193@KRBTEST.COM'} at 127.12.201.193:51093
I20250903 08:07:55.367470 14925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:55.360817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.193:38593 (local address 127.12.201.254:36815)
0903 08:07:55.360931 (+ 114us) server_negotiation.cc:207] Beginning negotiation
0903 08:07:55.360935 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:07:55.361151 (+ 216us) server_negotiation.cc:408] Connection header received
0903 08:07:55.361327 (+ 176us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:07:55.361334 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:07:55.361387 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:07:55.361466 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:07:55.362159 (+ 693us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:55.362891 (+ 732us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:07:55.363663 (+ 772us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:55.363873 (+ 210us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:55.364651 (+ 778us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:07:55.364667 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:07:55.364669 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:07:55.364688 (+ 19us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:07:55.366109 (+ 1421us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:07:55.366474 (+ 365us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:07:55.366477 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:07:55.366479 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:07:55.366516 (+ 37us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:07:55.366721 (+ 205us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:07:55.366724 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:07:55.366726 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:07:55.366850 (+ 124us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:07:55.366953 (+ 103us) server_negotiation.cc:1036] Waiting for connection context
0903 08:07:55.367207 (+ 254us) server_negotiation.cc:300] Negotiation successful
0903 08:07:55.367319 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20250903 08:07:55.367488 15086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:55.360771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:36815 (local address 127.12.201.193:38593)
0903 08:07:55.361080 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0903 08:07:55.361093 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0903 08:07:55.361193 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:07:55.361567 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:07:55.361570 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:07:55.361762 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:07:55.362029 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:55.362039 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:55.363023 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:07:55.363028 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:07:55.363553 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:55.363561 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:55.363662 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:55.364257 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:07:55.364269 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:07:55.364543 (+ 274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:07:55.366211 (+ 1668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:07:55.366215 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:07:55.366217 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:07:55.366386 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:07:55.366594 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:07:55.366599 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:07:55.366601 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:07:55.366644 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:07:55.366951 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:07:55.366954 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:07:55.367057 (+ 103us) client_negotiation.cc:769] Sending connection context
0903 08:07:55.367150 (+ 93us) client_negotiation.cc:240] Negotiation successful
0903 08:07:55.367306 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":91,"threads_started":1}
WARNING: no policy specified for kudu/127.12.201.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.201.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.201.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.201.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.201.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.201.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.201.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.201.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/krb5kdc/kudu.keytab.
I20250903 08:07:55.406500 13095 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUML90r/build/debug/bin/kudu
/tmp/dist-test-taskUML90r/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.201.194:0
--local_ip_for_outbound_sockets=127.12.201.194
--webserver_interface=127.12.201.194
--webserver_port=0
--tserver_master_addrs=127.12.201.254:36815
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.201.212:45945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 08:07:55.507859 15091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:07:55.508072 15091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:07:55.508159 15091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:07:55.511324 15091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 08:07:55.511381 15091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:07:55.511452 15091 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20250903 08:07:55.511490 15091 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20250903 08:07:55.511554 15091 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.201.194
I20250903 08:07:55.515575 15091 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.201.212:45945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.201.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.201.194
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.201.254:36815
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.201.194
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 07:50:08 UTC on 5fd53c4cbb9d
build id 7874
I20250903 08:07:55.516546 15091 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:07:55.517351 15091 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:07:55.523398 15099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:07:55.523414 15096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:07:55.523576 15091 server_base.cc:1047] running on GCE node
W20250903 08:07:55.523417 15097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:07:55.523978 15091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:07:55.524503 15091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:07:55.525655 15091 hybrid_clock.cc:648] HybridClock initialized: now 1756886875525632 us; error 35 us; skew 500 ppm
Sep 03 08:07:55 dist-test-slave-27r9 krb5kdc[14827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886875, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 08:07:55.528208 15091 init.cc:377] Logged in from keytab as kudu/127.12.201.194@KRBTEST.COM (short username kudu)
I20250903 08:07:55.529160 15091 webserver.cc:480] Webserver started at http://127.12.201.194:40261/ using document root <none> and password file <none>
I20250903 08:07:55.529675 15091 fs_manager.cc:362] Metadata directory not provided
I20250903 08:07:55.529728 15091 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:07:55.529902 15091 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:07:55.531565 15091 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/minicluster-data/ts-1/data/instance:
uuid: "18892f6d43244679a593d7aad5877a09"
format_stamp: "Formatted at 2025-09-03 08:07:55 on dist-test-slave-27r9"
server_key: "2a88620a095c32b747f4561e19741623"
server_key_iv: "81035c43e9c26440a637274cdb1b5ad5"
server_key_version: "encryptionkey@0"
I20250903 08:07:55.532006 15091 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/minicluster-data/ts-1/wal/instance:
uuid: "18892f6d43244679a593d7aad5877a09"
format_stamp: "Formatted at 2025-09-03 08:07:55 on dist-test-slave-27r9"
server_key: "2a88620a095c32b747f4561e19741623"
server_key_iv: "81035c43e9c26440a637274cdb1b5ad5"
server_key_version: "encryptionkey@0"
I20250903 08:07:55.535092 15091 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250903 08:07:55.537037 15106 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:07:55.537897 15091 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250903 08:07:55.537995 15091 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/minicluster-data/ts-1/wal
uuid: "18892f6d43244679a593d7aad5877a09"
format_stamp: "Formatted at 2025-09-03 08:07:55 on dist-test-slave-27r9"
server_key: "2a88620a095c32b747f4561e19741623"
server_key_iv: "81035c43e9c26440a637274cdb1b5ad5"
server_key_version: "encryptionkey@0"
I20250903 08:07:55.538076 15091 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:07:55.565624 15091 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:07:55.568356 15091 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:07:55.568533 15091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:07:55.569878 15091 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 08:07:55.569929 15091 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 08:07:55.569994 15091 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 08:07:55.570024 15091 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 03 08:07:55 dist-test-slave-27r9 krb5kdc[14827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886875, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.194@KRBTEST.COM for kudu/127.12.201.254@KRBTEST.COM
I20250903 08:07:55.580940 15091 rpc_server.cc:307] RPC server started. Bound to: 127.12.201.194:45871
I20250903 08:07:55.580991 15221 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.201.194:45871 every 8 connection(s)
I20250903 08:07:55.581204 14925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:55.570551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.194:34103 (local address 127.12.201.254:36815)
0903 08:07:55.570665 (+ 114us) server_negotiation.cc:207] Beginning negotiation
0903 08:07:55.570669 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:07:55.571501 (+ 832us) server_negotiation.cc:408] Connection header received
0903 08:07:55.572393 (+ 892us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:07:55.572397 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:07:55.572458 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:07:55.572537 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:07:55.573840 (+ 1303us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:55.574419 (+ 579us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:07:55.575125 (+ 706us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:55.575303 (+ 178us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:55.577795 (+ 2492us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:07:55.577815 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:07:55.577818 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:07:55.577847 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:07:55.579331 (+ 1484us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:07:55.579785 (+ 454us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:07:55.579792 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:07:55.579796 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:07:55.579856 (+ 60us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:07:55.580120 (+ 264us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:07:55.580124 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:07:55.580125 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:07:55.580318 (+ 193us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:07:55.580432 (+ 114us) server_negotiation.cc:1036] Waiting for connection context
0903 08:07:55.580918 (+ 486us) server_negotiation.cc:300] Negotiation successful
0903 08:07:55.581035 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20250903 08:07:55.581854 15115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:55.570877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:36815 (local address 127.12.201.194:34103)
0903 08:07:55.571355 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0903 08:07:55.571391 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0903 08:07:55.572193 (+ 802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:07:55.572721 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:07:55.572731 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:07:55.573147 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:07:55.573697 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:55.573709 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:55.574539 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:07:55.574545 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:07:55.574972 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:55.575007 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:55.575206 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:55.575847 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:07:55.575884 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:07:55.577645 (+ 1761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:07:55.579451 (+ 1806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:07:55.579457 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:07:55.579468 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:07:55.579684 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:07:55.579950 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:07:55.579953 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:07:55.579954 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:07:55.580031 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:07:55.580431 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:07:55.580436 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:07:55.580643 (+ 207us) client_negotiation.cc:769] Sending connection context
0903 08:07:55.580871 (+ 228us) client_negotiation.cc:240] Negotiation successful
0903 08:07:55.581145 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":142,"threads_started":1}
I20250903 08:07:55.582089 15091 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/minicluster-data/ts-1/data/info.pb
I20250903 08:07:55.583499 15222 heartbeater.cc:344] Connected to a master server at 127.12.201.254:36815
I20250903 08:07:55.583647 15222 heartbeater.cc:461] Registering TS with master...
I20250903 08:07:55.584180 15222 heartbeater.cc:507] Master 127.12.201.254:36815 requested a full tablet report, sending...
I20250903 08:07:55.585050 14875 ts_manager.cc:194] Registered new tserver with Master: 18892f6d43244679a593d7aad5877a09 (127.12.201.194:45871)
I20250903 08:07:55.585777 14875 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.201.194@KRBTEST.COM'} at 127.12.201.194:34103
I20250903 08:07:55.590560 14925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:55.584972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.194:55855 (local address 127.12.201.254:36815)
0903 08:07:55.585097 (+ 125us) server_negotiation.cc:207] Beginning negotiation
0903 08:07:55.585112 (+ 15us) server_negotiation.cc:400] Waiting for connection header
0903 08:07:55.585334 (+ 222us) server_negotiation.cc:408] Connection header received
0903 08:07:55.585440 (+ 106us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:07:55.585443 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:07:55.585479 (+ 36us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:07:55.585541 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:07:55.586132 (+ 591us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:55.586590 (+ 458us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:07:55.587217 (+ 627us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:55.587429 (+ 212us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:55.588113 (+ 684us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:07:55.588129 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:07:55.588132 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:07:55.588151 (+ 19us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:07:55.589235 (+ 1084us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:07:55.589590 (+ 355us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:07:55.589594 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:07:55.589595 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:07:55.589633 (+ 38us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:07:55.589843 (+ 210us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:07:55.589846 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:07:55.589848 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:07:55.589992 (+ 144us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:07:55.590093 (+ 101us) server_negotiation.cc:1036] Waiting for connection context
0903 08:07:55.590325 (+ 232us) server_negotiation.cc:300] Negotiation successful
0903 08:07:55.590434 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20250903 08:07:55.590569 15228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:55.584904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:36815 (local address 127.12.201.194:55855)
0903 08:07:55.585251 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0903 08:07:55.585266 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0903 08:07:55.585363 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:07:55.585570 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:07:55.585573 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:07:55.585754 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:07:55.586020 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:55.586031 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:55.586690 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:07:55.586693 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:07:55.587118 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:55.587126 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:55.587229 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:55.587724 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:07:55.587736 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:07:55.587999 (+ 263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:07:55.589344 (+ 1345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:07:55.589347 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:07:55.589350 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:07:55.589504 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:07:55.589716 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:07:55.589721 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:07:55.589723 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:07:55.589765 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:07:55.590093 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:07:55.590097 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:07:55.590178 (+ 81us) client_negotiation.cc:769] Sending connection context
0903 08:07:55.590267 (+ 89us) client_negotiation.cc:240] Negotiation successful
0903 08:07:55.590390 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":83,"threads_started":1}
I20250903 08:07:55.591981 13095 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUML90r/build/debug/bin/kudu as pid 15091
I20250903 08:07:55.592103 13095 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/minicluster-data/ts-1/wal/instance
I20250903 08:07:55.592394 13095 external_mini_cluster.cc:1468] Setting key 00a248202376189d6dde7c34335e3c09
WARNING: no policy specified for kudu/127.12.201.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.201.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.201.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.201.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.201.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.201.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.201.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.201.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/krb5kdc/kudu.keytab.
I20250903 08:07:55.637435 13095 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUML90r/build/debug/bin/kudu
/tmp/dist-test-taskUML90r/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.201.195:0
--local_ip_for_outbound_sockets=127.12.201.195
--webserver_interface=127.12.201.195
--webserver_port=0
--tserver_master_addrs=127.12.201.254:36815
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.201.212:45945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 08:07:55.741463 15233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:07:55.741706 15233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:07:55.741792 15233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:07:55.745066 15233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 08:07:55.745131 15233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:07:55.745230 15233 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20250903 08:07:55.745260 15233 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20250903 08:07:55.745330 15233 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.201.195
I20250903 08:07:55.749416 15233 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.201.212:45945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.201.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.201.195
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.201.254:36815
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15233
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.201.195
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 07:50:08 UTC on 5fd53c4cbb9d
build id 7874
I20250903 08:07:55.750433 15233 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:07:55.751276 15233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:07:55.757483 15239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:07:55.757477 15241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:07:55.757575 15233 server_base.cc:1047] running on GCE node
W20250903 08:07:55.757476 15238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:07:55.758020 15233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:07:55.758513 15233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:07:55.759650 15233 hybrid_clock.cc:648] HybridClock initialized: now 1756886875759637 us; error 33 us; skew 500 ppm
Sep 03 08:07:55 dist-test-slave-27r9 krb5kdc[14827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886875, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 08:07:55.762224 15233 init.cc:377] Logged in from keytab as kudu/127.12.201.195@KRBTEST.COM (short username kudu)
I20250903 08:07:55.763204 15233 webserver.cc:480] Webserver started at http://127.12.201.195:36867/ using document root <none> and password file <none>
I20250903 08:07:55.763700 15233 fs_manager.cc:362] Metadata directory not provided
I20250903 08:07:55.763746 15233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:07:55.763890 15233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:07:55.765457 15233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/minicluster-data/ts-2/data/instance:
uuid: "c35d28b86cb846d399e9beb0a914d1d4"
format_stamp: "Formatted at 2025-09-03 08:07:55 on dist-test-slave-27r9"
server_key: "bf2e25bc709969c6cdf8e2b494a400c0"
server_key_iv: "c41b38e3be47a69d78c16d9d9c64373d"
server_key_version: "encryptionkey@0"
I20250903 08:07:55.765861 15233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/minicluster-data/ts-2/wal/instance:
uuid: "c35d28b86cb846d399e9beb0a914d1d4"
format_stamp: "Formatted at 2025-09-03 08:07:55 on dist-test-slave-27r9"
server_key: "bf2e25bc709969c6cdf8e2b494a400c0"
server_key_iv: "c41b38e3be47a69d78c16d9d9c64373d"
server_key_version: "encryptionkey@0"
I20250903 08:07:55.768987 15233 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20250903 08:07:55.770931 15248 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:07:55.771859 15233 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250903 08:07:55.771944 15233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/minicluster-data/ts-2/wal
uuid: "c35d28b86cb846d399e9beb0a914d1d4"
format_stamp: "Formatted at 2025-09-03 08:07:55 on dist-test-slave-27r9"
server_key: "bf2e25bc709969c6cdf8e2b494a400c0"
server_key_iv: "c41b38e3be47a69d78c16d9d9c64373d"
server_key_version: "encryptionkey@0"
I20250903 08:07:55.772017 15233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:07:55.793107 15233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:07:55.795593 15233 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:07:55.795732 15233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:07:55.797008 15233 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 08:07:55.797058 15233 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 08:07:55.797107 15233 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 08:07:55.797140 15233 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 03 08:07:55 dist-test-slave-27r9 krb5kdc[14827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886875, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.195@KRBTEST.COM for kudu/127.12.201.254@KRBTEST.COM
I20250903 08:07:55.808000 15233 rpc_server.cc:307] RPC server started. Bound to: 127.12.201.195:41255
I20250903 08:07:55.808043 15363 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.201.195:41255 every 8 connection(s)
I20250903 08:07:55.808929 14925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:55.797596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.195:46735 (local address 127.12.201.254:36815)
0903 08:07:55.797709 (+ 113us) server_negotiation.cc:207] Beginning negotiation
0903 08:07:55.797713 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:07:55.798482 (+ 769us) server_negotiation.cc:408] Connection header received
0903 08:07:55.799339 (+ 857us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:07:55.799343 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:07:55.799416 (+ 73us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:07:55.799504 (+ 88us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:07:55.800830 (+ 1326us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:55.801394 (+ 564us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:07:55.802327 (+ 933us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:55.802545 (+ 218us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:55.805389 (+ 2844us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:07:55.805407 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:07:55.805409 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:07:55.805433 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:07:55.806968 (+ 1535us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:07:55.807529 (+ 561us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:07:55.807534 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:07:55.807538 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:07:55.807600 (+ 62us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:07:55.807909 (+ 309us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:07:55.807911 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:07:55.807913 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:07:55.808133 (+ 220us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:07:55.808233 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
0903 08:07:55.808720 (+ 487us) server_negotiation.cc:300] Negotiation successful
0903 08:07:55.808814 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20250903 08:07:55.809070 15233 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/minicluster-data/ts-2/data/info.pb
I20250903 08:07:55.809821 15257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:55.797911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:36815 (local address 127.12.201.195:46735)
0903 08:07:55.798352 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0903 08:07:55.798388 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0903 08:07:55.799147 (+ 759us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:07:55.799672 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:07:55.799681 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:07:55.800163 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:07:55.800678 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:55.800690 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:55.801527 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:07:55.801534 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:07:55.802196 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:55.802208 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:55.802427 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:55.803222 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:07:55.803259 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:07:55.805226 (+ 1967us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:07:55.807093 (+ 1867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:07:55.807099 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:07:55.807108 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:07:55.807431 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:07:55.807695 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:07:55.807698 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:07:55.807700 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:07:55.807819 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:07:55.808234 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:07:55.808240 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:07:55.808487 (+ 247us) client_negotiation.cc:769] Sending connection context
0903 08:07:55.808696 (+ 209us) client_negotiation.cc:240] Negotiation successful
0903 08:07:55.808943 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":122,"threads_started":1}
I20250903 08:07:55.810885 15364 heartbeater.cc:344] Connected to a master server at 127.12.201.254:36815
I20250903 08:07:55.811059 15364 heartbeater.cc:461] Registering TS with master...
I20250903 08:07:55.811537 15364 heartbeater.cc:507] Master 127.12.201.254:36815 requested a full tablet report, sending...
I20250903 08:07:55.812487 14874 ts_manager.cc:194] Registered new tserver with Master: c35d28b86cb846d399e9beb0a914d1d4 (127.12.201.195:41255)
I20250903 08:07:55.813107 13095 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUML90r/build/debug/bin/kudu as pid 15233
I20250903 08:07:55.813148 14874 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.201.195@KRBTEST.COM'} at 127.12.201.195:46735
I20250903 08:07:55.813223 13095 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/minicluster-data/ts-2/wal/instance
I20250903 08:07:55.813482 13095 external_mini_cluster.cc:1468] Setting key 95040f965ab343ece7d2c89ebe8e2aea
I20250903 08:07:55.815383 13095 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250903 08:07:55.819319 14925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:55.812853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.195:39599 (local address 127.12.201.254:36815)
0903 08:07:55.812980 (+ 127us) server_negotiation.cc:207] Beginning negotiation
0903 08:07:55.812984 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:07:55.813128 (+ 144us) server_negotiation.cc:408] Connection header received
0903 08:07:55.813267 (+ 139us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:07:55.813270 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:07:55.813312 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:07:55.813379 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:07:55.814249 (+ 870us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:55.814830 (+ 581us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:07:55.815598 (+ 768us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:55.815724 (+ 126us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:55.816619 (+ 895us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:07:55.816633 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:07:55.816635 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:07:55.816655 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:07:55.817947 (+ 1292us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:07:55.818388 (+ 441us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:07:55.818390 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:07:55.818391 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:07:55.818424 (+ 33us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:07:55.818638 (+ 214us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:07:55.818640 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:07:55.818641 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:07:55.818752 (+ 111us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:07:55.818817 (+ 65us) server_negotiation.cc:1036] Waiting for connection context
0903 08:07:55.819125 (+ 308us) server_negotiation.cc:300] Negotiation successful
0903 08:07:55.819210 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20250903 08:07:55.819433 15370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:55.812796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:36815 (local address 127.12.201.195:39599)
0903 08:07:55.813055 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0903 08:07:55.813066 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0903 08:07:55.813154 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:07:55.813706 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:07:55.813709 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:07:55.813872 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:07:55.814105 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:55.814116 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:55.814938 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:07:55.814941 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:07:55.815497 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:55.815506 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:55.815610 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:55.816089 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:07:55.816104 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:07:55.816501 (+ 397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:07:55.818040 (+ 1539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:07:55.818046 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:07:55.818051 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:07:55.818278 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:07:55.818494 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:07:55.818500 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:07:55.818505 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:07:55.818566 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:07:55.818829 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:07:55.818835 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:07:55.818930 (+ 95us) client_negotiation.cc:769] Sending connection context
0903 08:07:55.819076 (+ 146us) client_negotiation.cc:240] Negotiation successful
0903 08:07:55.819216 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":74,"threads_started":1}
Sep 03 08:07:55 dist-test-slave-27r9 krb5kdc[14827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 03 08:07:55 dist-test-slave-27r9 krb5kdc[14827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.201.254@KRBTEST.COM
I20250903 08:07:55.838253 14925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:55.831344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53140 (local address 127.12.201.254:36815)
0903 08:07:55.831491 (+ 147us) server_negotiation.cc:207] Beginning negotiation
0903 08:07:55.831495 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:07:55.831508 (+ 13us) server_negotiation.cc:408] Connection header received
0903 08:07:55.831547 (+ 39us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:07:55.831549 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:07:55.831587 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:07:55.831656 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:07:55.832484 (+ 828us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:55.832985 (+ 501us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:07:55.833623 (+ 638us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:55.833833 (+ 210us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:55.835863 (+ 2030us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:07:55.835881 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:07:55.835883 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:07:55.835907 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:07:55.837119 (+ 1212us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:07:55.837459 (+ 340us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:07:55.837462 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:07:55.837463 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:07:55.837496 (+ 33us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:07:55.837648 (+ 152us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:07:55.837650 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:07:55.837651 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:07:55.837777 (+ 126us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:07:55.837874 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
0903 08:07:55.838042 (+ 168us) server_negotiation.cc:300] Negotiation successful
0903 08:07:55.838135 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20250903 08:07:55.840811 14874 catalog_manager.cc:2232] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53140:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 08:07:55.843268 14874 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 08:07:55.854841 15381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:55.850924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.193:39419 (local address 127.0.0.1:51148)
0903 08:07:55.851267 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0903 08:07:55.851303 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0903 08:07:55.851430 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:07:55.852114 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:07:55.852119 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:07:55.852145 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 08:07:55.852439 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:55.852446 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:55.853606 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:07:55.853609 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:07:55.854342 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:55.854350 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:55.854455 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:55.854508 (+ 53us) client_negotiation.cc:769] Sending connection context
0903 08:07:55.854595 (+ 87us) client_negotiation.cc:240] Negotiation successful
0903 08:07:55.854690 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":113,"threads_started":1}
I20250903 08:07:55.856149 15382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:55.851120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51148 (local address 127.12.201.193:39419)
0903 08:07:55.851364 (+ 244us) server_negotiation.cc:207] Beginning negotiation
0903 08:07:55.851369 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 08:07:55.851397 (+ 28us) server_negotiation.cc:408] Connection header received
0903 08:07:55.851585 (+ 188us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:07:55.851594 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:07:55.851777 (+ 183us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:07:55.851937 (+ 160us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 08:07:55.852565 (+ 628us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:55.853494 (+ 929us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:07:55.855207 (+ 1713us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:55.855735 (+ 528us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:55.855844 (+ 109us) server_negotiation.cc:1036] Waiting for connection context
0903 08:07:55.855904 (+ 60us) server_negotiation.cc:300] Negotiation successful
0903 08:07:55.856000 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":66,"threads_started":1}
I20250903 08:07:55.856444 15385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:55.851939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60350 (local address 127.12.201.195:41255)
0903 08:07:55.852688 (+ 749us) server_negotiation.cc:207] Beginning negotiation
0903 08:07:55.852693 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 08:07:55.852710 (+ 17us) server_negotiation.cc:408] Connection header received
0903 08:07:55.852770 (+ 60us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:07:55.852774 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:07:55.852885 (+ 111us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:07:55.852989 (+ 104us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 08:07:55.853418 (+ 429us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:55.854491 (+ 1073us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:07:55.855619 (+ 1128us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:55.856072 (+ 453us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:55.856174 (+ 102us) server_negotiation.cc:1036] Waiting for connection context
0903 08:07:55.856233 (+ 59us) server_negotiation.cc:300] Negotiation successful
0903 08:07:55.856312 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":670,"thread_start_us":97,"threads_started":1}
I20250903 08:07:55.856462 15383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:55.851949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.194:45871 (local address 127.0.0.1:51956)
0903 08:07:55.852369 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0903 08:07:55.852382 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0903 08:07:55.852469 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:07:55.853582 (+ 1113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:07:55.853585 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:07:55.853608 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 08:07:55.853827 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:55.853832 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:55.855209 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:07:55.855212 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:07:55.856053 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:55.856063 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:55.856190 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:55.856210 (+ 20us) client_negotiation.cc:769] Sending connection context
0903 08:07:55.856284 (+ 74us) client_negotiation.cc:240] Negotiation successful
0903 08:07:55.856336 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":99,"threads_started":1}
I20250903 08:07:55.857059 15386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:55.852332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51956 (local address 127.12.201.194:45871)
0903 08:07:55.853224 (+ 892us) server_negotiation.cc:207] Beginning negotiation
0903 08:07:55.853228 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:07:55.853245 (+ 17us) server_negotiation.cc:408] Connection header received
0903 08:07:55.853304 (+ 59us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:07:55.853307 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:07:55.853440 (+ 133us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:07:55.853590 (+ 150us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 08:07:55.853936 (+ 346us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:55.855087 (+ 1151us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:07:55.856216 (+ 1129us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:55.856740 (+ 524us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:55.856839 (+ 99us) server_negotiation.cc:1036] Waiting for connection context
0903 08:07:55.856892 (+ 53us) server_negotiation.cc:300] Negotiation successful
0903 08:07:55.856955 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":813,"thread_start_us":99,"threads_started":1}
I20250903 08:07:55.855845 15384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:55.851649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.195:41255 (local address 127.0.0.1:60350)
0903 08:07:55.852186 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0903 08:07:55.852200 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0903 08:07:55.852439 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:07:55.853007 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:07:55.853010 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:07:55.853030 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 08:07:55.853288 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:55.853295 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:55.854656 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:07:55.854659 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:07:55.855482 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:55.855490 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:55.855593 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:55.855616 (+ 23us) client_negotiation.cc:769] Sending connection context
0903 08:07:55.855667 (+ 51us) client_negotiation.cc:240] Negotiation successful
0903 08:07:55.855725 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":100,"threads_started":1}
I20250903 08:07:55.859413 15014 tablet_service.cc:1468] Processing CreateTablet for tablet 85e451aecb5a461daae1bd11253031da (DEFAULT_TABLE table=test-table [id=9eac45e9bea44060bce09117a2dfcdce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:07:55.859524 15156 tablet_service.cc:1468] Processing CreateTablet for tablet 85e451aecb5a461daae1bd11253031da (DEFAULT_TABLE table=test-table [id=9eac45e9bea44060bce09117a2dfcdce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:07:55.859524 15298 tablet_service.cc:1468] Processing CreateTablet for tablet 85e451aecb5a461daae1bd11253031da (DEFAULT_TABLE table=test-table [id=9eac45e9bea44060bce09117a2dfcdce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:07:55.860289 15014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85e451aecb5a461daae1bd11253031da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:07:55.860399 15298 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85e451aecb5a461daae1bd11253031da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:07:55.860399 15156 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85e451aecb5a461daae1bd11253031da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:07:55.865549 15387 tablet_bootstrap.cc:492] T 85e451aecb5a461daae1bd11253031da P 4fe0343417e4407cacb543879bd8a7e9: Bootstrap starting.
I20250903 08:07:55.866818 15389 tablet_bootstrap.cc:492] T 85e451aecb5a461daae1bd11253031da P c35d28b86cb846d399e9beb0a914d1d4: Bootstrap starting.
I20250903 08:07:55.866883 15388 tablet_bootstrap.cc:492] T 85e451aecb5a461daae1bd11253031da P 18892f6d43244679a593d7aad5877a09: Bootstrap starting.
I20250903 08:07:55.868558 15389 tablet_bootstrap.cc:654] T 85e451aecb5a461daae1bd11253031da P c35d28b86cb846d399e9beb0a914d1d4: Neither blocks nor log segments found. Creating new log.
I20250903 08:07:55.868610 15387 tablet_bootstrap.cc:654] T 85e451aecb5a461daae1bd11253031da P 4fe0343417e4407cacb543879bd8a7e9: Neither blocks nor log segments found. Creating new log.
I20250903 08:07:55.868790 15388 tablet_bootstrap.cc:654] T 85e451aecb5a461daae1bd11253031da P 18892f6d43244679a593d7aad5877a09: Neither blocks nor log segments found. Creating new log.
I20250903 08:07:55.869511 15387 log.cc:826] T 85e451aecb5a461daae1bd11253031da P 4fe0343417e4407cacb543879bd8a7e9: Log is configured to *not* fsync() on all Append() calls
I20250903 08:07:55.869555 15389 log.cc:826] T 85e451aecb5a461daae1bd11253031da P c35d28b86cb846d399e9beb0a914d1d4: Log is configured to *not* fsync() on all Append() calls
I20250903 08:07:55.869907 15388 log.cc:826] T 85e451aecb5a461daae1bd11253031da P 18892f6d43244679a593d7aad5877a09: Log is configured to *not* fsync() on all Append() calls
I20250903 08:07:55.871237 15387 tablet_bootstrap.cc:492] T 85e451aecb5a461daae1bd11253031da P 4fe0343417e4407cacb543879bd8a7e9: No bootstrap required, opened a new log
I20250903 08:07:55.871421 15387 ts_tablet_manager.cc:1397] T 85e451aecb5a461daae1bd11253031da P 4fe0343417e4407cacb543879bd8a7e9: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20250903 08:07:55.871580 15388 tablet_bootstrap.cc:492] T 85e451aecb5a461daae1bd11253031da P 18892f6d43244679a593d7aad5877a09: No bootstrap required, opened a new log
I20250903 08:07:55.871580 15389 tablet_bootstrap.cc:492] T 85e451aecb5a461daae1bd11253031da P c35d28b86cb846d399e9beb0a914d1d4: No bootstrap required, opened a new log
I20250903 08:07:55.871783 15388 ts_tablet_manager.cc:1397] T 85e451aecb5a461daae1bd11253031da P 18892f6d43244679a593d7aad5877a09: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20250903 08:07:55.872220 15389 ts_tablet_manager.cc:1397] T 85e451aecb5a461daae1bd11253031da P c35d28b86cb846d399e9beb0a914d1d4: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20250903 08:07:55.873785 15387 raft_consensus.cc:357] T 85e451aecb5a461daae1bd11253031da P 4fe0343417e4407cacb543879bd8a7e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe0343417e4407cacb543879bd8a7e9" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 39419 } } peers { permanent_uuid: "c35d28b86cb846d399e9beb0a914d1d4" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 41255 } } peers { permanent_uuid: "18892f6d43244679a593d7aad5877a09" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 45871 } }
I20250903 08:07:55.873962 15387 raft_consensus.cc:383] T 85e451aecb5a461daae1bd11253031da P 4fe0343417e4407cacb543879bd8a7e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:07:55.874018 15387 raft_consensus.cc:738] T 85e451aecb5a461daae1bd11253031da P 4fe0343417e4407cacb543879bd8a7e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fe0343417e4407cacb543879bd8a7e9, State: Initialized, Role: FOLLOWER
I20250903 08:07:55.874397 15387 consensus_queue.cc:260] T 85e451aecb5a461daae1bd11253031da P 4fe0343417e4407cacb543879bd8a7e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe0343417e4407cacb543879bd8a7e9" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 39419 } } peers { permanent_uuid: "c35d28b86cb846d399e9beb0a914d1d4" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 41255 } } peers { permanent_uuid: "18892f6d43244679a593d7aad5877a09" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 45871 } }
I20250903 08:07:55.874662 15388 raft_consensus.cc:357] T 85e451aecb5a461daae1bd11253031da P 18892f6d43244679a593d7aad5877a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe0343417e4407cacb543879bd8a7e9" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 39419 } } peers { permanent_uuid: "c35d28b86cb846d399e9beb0a914d1d4" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 41255 } } peers { permanent_uuid: "18892f6d43244679a593d7aad5877a09" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 45871 } }
I20250903 08:07:55.874871 15388 raft_consensus.cc:383] T 85e451aecb5a461daae1bd11253031da P 18892f6d43244679a593d7aad5877a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:07:55.874929 15388 raft_consensus.cc:738] T 85e451aecb5a461daae1bd11253031da P 18892f6d43244679a593d7aad5877a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18892f6d43244679a593d7aad5877a09, State: Initialized, Role: FOLLOWER
I20250903 08:07:55.874998 15389 raft_consensus.cc:357] T 85e451aecb5a461daae1bd11253031da P c35d28b86cb846d399e9beb0a914d1d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe0343417e4407cacb543879bd8a7e9" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 39419 } } peers { permanent_uuid: "c35d28b86cb846d399e9beb0a914d1d4" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 41255 } } peers { permanent_uuid: "18892f6d43244679a593d7aad5877a09" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 45871 } }
I20250903 08:07:55.875118 15080 heartbeater.cc:499] Master 127.12.201.254:36815 was elected leader, sending a full tablet report...
I20250903 08:07:55.875173 15389 raft_consensus.cc:383] T 85e451aecb5a461daae1bd11253031da P c35d28b86cb846d399e9beb0a914d1d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:07:55.875226 15389 raft_consensus.cc:738] T 85e451aecb5a461daae1bd11253031da P c35d28b86cb846d399e9beb0a914d1d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c35d28b86cb846d399e9beb0a914d1d4, State: Initialized, Role: FOLLOWER
I20250903 08:07:55.875306 15387 ts_tablet_manager.cc:1428] T 85e451aecb5a461daae1bd11253031da P 4fe0343417e4407cacb543879bd8a7e9: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250903 08:07:55.875353 15388 consensus_queue.cc:260] T 85e451aecb5a461daae1bd11253031da P 18892f6d43244679a593d7aad5877a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe0343417e4407cacb543879bd8a7e9" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 39419 } } peers { permanent_uuid: "c35d28b86cb846d399e9beb0a914d1d4" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 41255 } } peers { permanent_uuid: "18892f6d43244679a593d7aad5877a09" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 45871 } }
I20250903 08:07:55.875624 15389 consensus_queue.cc:260] T 85e451aecb5a461daae1bd11253031da P c35d28b86cb846d399e9beb0a914d1d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe0343417e4407cacb543879bd8a7e9" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 39419 } } peers { permanent_uuid: "c35d28b86cb846d399e9beb0a914d1d4" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 41255 } } peers { permanent_uuid: "18892f6d43244679a593d7aad5877a09" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 45871 } }
I20250903 08:07:55.876034 15222 heartbeater.cc:499] Master 127.12.201.254:36815 was elected leader, sending a full tablet report...
I20250903 08:07:55.876158 15388 ts_tablet_manager.cc:1428] T 85e451aecb5a461daae1bd11253031da P 18892f6d43244679a593d7aad5877a09: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250903 08:07:55.876675 15364 heartbeater.cc:499] Master 127.12.201.254:36815 was elected leader, sending a full tablet report...
I20250903 08:07:55.877727 15389 ts_tablet_manager.cc:1428] T 85e451aecb5a461daae1bd11253031da P c35d28b86cb846d399e9beb0a914d1d4: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
W20250903 08:07:56.063933 15365 tablet.cc:2378] T 85e451aecb5a461daae1bd11253031da P c35d28b86cb846d399e9beb0a914d1d4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250903 08:07:56.083212 15223 tablet.cc:2378] T 85e451aecb5a461daae1bd11253031da P 18892f6d43244679a593d7aad5877a09: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250903 08:07:56.113631 15081 tablet.cc:2378] T 85e451aecb5a461daae1bd11253031da P 4fe0343417e4407cacb543879bd8a7e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250903 08:07:56.137209 15395 raft_consensus.cc:491] T 85e451aecb5a461daae1bd11253031da P c35d28b86cb846d399e9beb0a914d1d4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 08:07:56.137441 15395 raft_consensus.cc:513] T 85e451aecb5a461daae1bd11253031da P c35d28b86cb846d399e9beb0a914d1d4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe0343417e4407cacb543879bd8a7e9" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 39419 } } peers { permanent_uuid: "c35d28b86cb846d399e9beb0a914d1d4" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 41255 } } peers { permanent_uuid: "18892f6d43244679a593d7aad5877a09" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 45871 } }
I20250903 08:07:56.138695 15395 leader_election.cc:290] T 85e451aecb5a461daae1bd11253031da P c35d28b86cb846d399e9beb0a914d1d4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4fe0343417e4407cacb543879bd8a7e9 (127.12.201.193:39419), 18892f6d43244679a593d7aad5877a09 (127.12.201.194:45871)
I20250903 08:07:56.141657 15257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:56.138889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.193:39419 (local address 127.12.201.195:38859)
0903 08:07:56.139042 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0903 08:07:56.139060 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0903 08:07:56.139146 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:07:56.139461 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:07:56.139464 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:07:56.139489 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 08:07:56.139736 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:56.139745 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:56.140651 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:07:56.140655 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:07:56.141325 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:56.141333 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:56.141425 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:56.141448 (+ 23us) client_negotiation.cc:769] Sending connection context
0903 08:07:56.141497 (+ 49us) client_negotiation.cc:240] Negotiation successful
0903 08:07:56.141546 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20250903 08:07:56.142191 15382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:56.139002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.195:38859 (local address 127.12.201.193:39419)
0903 08:07:56.139109 (+ 107us) server_negotiation.cc:207] Beginning negotiation
0903 08:07:56.139113 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:07:56.139129 (+ 16us) server_negotiation.cc:408] Connection header received
0903 08:07:56.139272 (+ 143us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:07:56.139275 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:07:56.139335 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:07:56.139419 (+ 84us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 08:07:56.139859 (+ 440us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:56.140532 (+ 673us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:07:56.141462 (+ 930us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:56.141936 (+ 474us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:56.141973 (+ 37us) server_negotiation.cc:1036] Waiting for connection context
0903 08:07:56.142024 (+ 51us) server_negotiation.cc:300] Negotiation successful
0903 08:07:56.142077 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20250903 08:07:56.142647 15396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:56.139181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.194:45871 (local address 127.12.201.195:51367)
0903 08:07:56.139542 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0903 08:07:56.139555 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0903 08:07:56.139634 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:07:56.139844 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:07:56.139847 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:07:56.139862 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 08:07:56.140170 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:56.140177 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:56.140947 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:07:56.140950 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:07:56.141612 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:56.141621 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:56.142313 (+ 692us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:56.142327 (+ 14us) client_negotiation.cc:769] Sending connection context
0903 08:07:56.142420 (+ 93us) client_negotiation.cc:240] Negotiation successful
0903 08:07:56.142489 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":121,"threads_started":1}
I20250903 08:07:56.142663 15386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:56.139296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.195:51367 (local address 127.12.201.194:45871)
0903 08:07:56.139466 (+ 170us) server_negotiation.cc:207] Beginning negotiation
0903 08:07:56.139470 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:07:56.139627 (+ 157us) server_negotiation.cc:408] Connection header received
0903 08:07:56.139706 (+ 79us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:07:56.139709 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:07:56.139755 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:07:56.139831 (+ 76us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 08:07:56.140290 (+ 459us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:56.140835 (+ 545us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:07:56.141752 (+ 917us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:56.142218 (+ 466us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:56.142256 (+ 38us) server_negotiation.cc:1036] Waiting for connection context
0903 08:07:56.142471 (+ 215us) server_negotiation.cc:300] Negotiation successful
0903 08:07:56.142552 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20250903 08:07:56.142818 15034 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "85e451aecb5a461daae1bd11253031da" candidate_uuid: "c35d28b86cb846d399e9beb0a914d1d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fe0343417e4407cacb543879bd8a7e9" is_pre_election: true
I20250903 08:07:56.143153 15034 raft_consensus.cc:2466] T 85e451aecb5a461daae1bd11253031da P 4fe0343417e4407cacb543879bd8a7e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c35d28b86cb846d399e9beb0a914d1d4 in term 0.
I20250903 08:07:56.143273 15176 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "85e451aecb5a461daae1bd11253031da" candidate_uuid: "c35d28b86cb846d399e9beb0a914d1d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18892f6d43244679a593d7aad5877a09" is_pre_election: true
I20250903 08:07:56.143519 15176 raft_consensus.cc:2466] T 85e451aecb5a461daae1bd11253031da P 18892f6d43244679a593d7aad5877a09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c35d28b86cb846d399e9beb0a914d1d4 in term 0.
I20250903 08:07:56.143577 15249 leader_election.cc:304] T 85e451aecb5a461daae1bd11253031da P c35d28b86cb846d399e9beb0a914d1d4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fe0343417e4407cacb543879bd8a7e9, c35d28b86cb846d399e9beb0a914d1d4; no voters:
I20250903 08:07:56.143797 15395 raft_consensus.cc:2802] T 85e451aecb5a461daae1bd11253031da P c35d28b86cb846d399e9beb0a914d1d4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 08:07:56.143868 15395 raft_consensus.cc:491] T 85e451aecb5a461daae1bd11253031da P c35d28b86cb846d399e9beb0a914d1d4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 08:07:56.143914 15395 raft_consensus.cc:3058] T 85e451aecb5a461daae1bd11253031da P c35d28b86cb846d399e9beb0a914d1d4 [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:07:56.144819 15395 raft_consensus.cc:513] T 85e451aecb5a461daae1bd11253031da P c35d28b86cb846d399e9beb0a914d1d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe0343417e4407cacb543879bd8a7e9" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 39419 } } peers { permanent_uuid: "c35d28b86cb846d399e9beb0a914d1d4" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 41255 } } peers { permanent_uuid: "18892f6d43244679a593d7aad5877a09" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 45871 } }
I20250903 08:07:56.145125 15395 leader_election.cc:290] T 85e451aecb5a461daae1bd11253031da P c35d28b86cb846d399e9beb0a914d1d4 [CANDIDATE]: Term 1 election: Requested vote from peers 4fe0343417e4407cacb543879bd8a7e9 (127.12.201.193:39419), 18892f6d43244679a593d7aad5877a09 (127.12.201.194:45871)
I20250903 08:07:56.145519 15034 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "85e451aecb5a461daae1bd11253031da" candidate_uuid: "c35d28b86cb846d399e9beb0a914d1d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fe0343417e4407cacb543879bd8a7e9"
I20250903 08:07:56.145541 15176 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "85e451aecb5a461daae1bd11253031da" candidate_uuid: "c35d28b86cb846d399e9beb0a914d1d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18892f6d43244679a593d7aad5877a09"
I20250903 08:07:56.145654 15034 raft_consensus.cc:3058] T 85e451aecb5a461daae1bd11253031da P 4fe0343417e4407cacb543879bd8a7e9 [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:07:56.145654 15176 raft_consensus.cc:3058] T 85e451aecb5a461daae1bd11253031da P 18892f6d43244679a593d7aad5877a09 [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:07:56.146878 15176 raft_consensus.cc:2466] T 85e451aecb5a461daae1bd11253031da P 18892f6d43244679a593d7aad5877a09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c35d28b86cb846d399e9beb0a914d1d4 in term 1.
I20250903 08:07:56.146878 15034 raft_consensus.cc:2466] T 85e451aecb5a461daae1bd11253031da P 4fe0343417e4407cacb543879bd8a7e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c35d28b86cb846d399e9beb0a914d1d4 in term 1.
I20250903 08:07:56.147238 15249 leader_election.cc:304] T 85e451aecb5a461daae1bd11253031da P c35d28b86cb846d399e9beb0a914d1d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18892f6d43244679a593d7aad5877a09, c35d28b86cb846d399e9beb0a914d1d4; no voters:
I20250903 08:07:56.147399 15395 raft_consensus.cc:2802] T 85e451aecb5a461daae1bd11253031da P c35d28b86cb846d399e9beb0a914d1d4 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 08:07:56.147616 15395 raft_consensus.cc:695] T 85e451aecb5a461daae1bd11253031da P c35d28b86cb846d399e9beb0a914d1d4 [term 1 LEADER]: Becoming Leader. State: Replica: c35d28b86cb846d399e9beb0a914d1d4, State: Running, Role: LEADER
I20250903 08:07:56.147905 15395 consensus_queue.cc:237] T 85e451aecb5a461daae1bd11253031da P c35d28b86cb846d399e9beb0a914d1d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe0343417e4407cacb543879bd8a7e9" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 39419 } } peers { permanent_uuid: "c35d28b86cb846d399e9beb0a914d1d4" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 41255 } } peers { permanent_uuid: "18892f6d43244679a593d7aad5877a09" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 45871 } }
I20250903 08:07:56.151050 14873 catalog_manager.cc:5582] T 85e451aecb5a461daae1bd11253031da P c35d28b86cb846d399e9beb0a914d1d4 reported cstate change: term changed from 0 to 1, leader changed from <none> to c35d28b86cb846d399e9beb0a914d1d4 (127.12.201.195). New cstate: current_term: 1 leader_uuid: "c35d28b86cb846d399e9beb0a914d1d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe0343417e4407cacb543879bd8a7e9" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 39419 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c35d28b86cb846d399e9beb0a914d1d4" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 41255 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "18892f6d43244679a593d7aad5877a09" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 45871 } health_report { overall_health: UNKNOWN } } }
Sep 03 08:07:56 dist-test-slave-27r9 krb5kdc[14827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886875, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.254@KRBTEST.COM for kudu/127.12.201.254@KRBTEST.COM
I20250903 08:07:56.235904 15404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:56.228603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:36815 (local address 127.0.0.1:53142)
0903 08:07:56.228876 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0903 08:07:56.228889 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0903 08:07:56.228989 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:07:56.229223 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:07:56.229225 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:07:56.229645 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:07:56.229839 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:56.229845 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:56.230562 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:07:56.230565 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:07:56.230919 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:56.230925 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:56.231042 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:56.231593 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:07:56.231605 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:07:56.232932 (+ 1327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:07:56.234594 (+ 1662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:07:56.234601 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:07:56.234606 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:07:56.234799 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:07:56.235031 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:07:56.235034 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:07:56.235035 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:07:56.235074 (+ 39us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:07:56.235381 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:07:56.235389 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:07:56.235476 (+ 87us) client_negotiation.cc:769] Sending connection context
0903 08:07:56.235572 (+ 96us) client_negotiation.cc:240] Negotiation successful
0903 08:07:56.235710 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":86,"threads_started":1}
I20250903 08:07:56.235904 14925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:56.228674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53142 (local address 127.12.201.254:36815)
0903 08:07:56.228785 (+ 111us) server_negotiation.cc:207] Beginning negotiation
0903 08:07:56.228789 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:07:56.228951 (+ 162us) server_negotiation.cc:408] Connection header received
0903 08:07:56.229063 (+ 112us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:07:56.229065 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:07:56.229112 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:07:56.229185 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:07:56.229927 (+ 742us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:56.230474 (+ 547us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:07:56.231048 (+ 574us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:56.231175 (+ 127us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:56.233046 (+ 1871us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:07:56.233060 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:07:56.233062 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:07:56.233089 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:07:56.234476 (+ 1387us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:07:56.234884 (+ 408us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:07:56.234888 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:07:56.234889 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:07:56.234928 (+ 39us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:07:56.235151 (+ 223us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:07:56.235154 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:07:56.235156 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:07:56.235281 (+ 125us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:07:56.235367 (+ 86us) server_negotiation.cc:1036] Waiting for connection context
0903 08:07:56.235626 (+ 259us) server_negotiation.cc:300] Negotiation successful
0903 08:07:56.235783 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":40}
I20250903 08:07:56.239557 14873 catalog_manager.cc:2232] Servicing CreateTable request from {username='kudu', principal='kudu/127.12.201.254@KRBTEST.COM'} at 127.0.0.1:53142:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20250903 08:07:56.240216 14873 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 08:07:56.244597 15156 tablet_service.cc:1468] Processing CreateTablet for tablet c524a3070fae4d03b2328a218fb73b18 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8e3229f11963443ca4a6dfc0820552dc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20250903 08:07:56.244562 15014 tablet_service.cc:1468] Processing CreateTablet for tablet c524a3070fae4d03b2328a218fb73b18 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8e3229f11963443ca4a6dfc0820552dc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20250903 08:07:56.244777 15298 tablet_service.cc:1468] Processing CreateTablet for tablet c524a3070fae4d03b2328a218fb73b18 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8e3229f11963443ca4a6dfc0820552dc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20250903 08:07:56.245198 15156 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c524a3070fae4d03b2328a218fb73b18. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:07:56.245198 15298 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c524a3070fae4d03b2328a218fb73b18. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:07:56.245214 15014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c524a3070fae4d03b2328a218fb73b18. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:07:56.247910 15389 tablet_bootstrap.cc:492] T c524a3070fae4d03b2328a218fb73b18 P c35d28b86cb846d399e9beb0a914d1d4: Bootstrap starting.
I20250903 08:07:56.248379 15387 tablet_bootstrap.cc:492] T c524a3070fae4d03b2328a218fb73b18 P 4fe0343417e4407cacb543879bd8a7e9: Bootstrap starting.
I20250903 08:07:56.248943 15389 tablet_bootstrap.cc:654] T c524a3070fae4d03b2328a218fb73b18 P c35d28b86cb846d399e9beb0a914d1d4: Neither blocks nor log segments found. Creating new log.
I20250903 08:07:56.249421 15387 tablet_bootstrap.cc:654] T c524a3070fae4d03b2328a218fb73b18 P 4fe0343417e4407cacb543879bd8a7e9: Neither blocks nor log segments found. Creating new log.
I20250903 08:07:56.249800 15388 tablet_bootstrap.cc:492] T c524a3070fae4d03b2328a218fb73b18 P 18892f6d43244679a593d7aad5877a09: Bootstrap starting.
I20250903 08:07:56.250496 15389 tablet_bootstrap.cc:492] T c524a3070fae4d03b2328a218fb73b18 P c35d28b86cb846d399e9beb0a914d1d4: No bootstrap required, opened a new log
I20250903 08:07:56.250582 15389 ts_tablet_manager.cc:1397] T c524a3070fae4d03b2328a218fb73b18 P c35d28b86cb846d399e9beb0a914d1d4: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20250903 08:07:56.250811 15388 tablet_bootstrap.cc:654] T c524a3070fae4d03b2328a218fb73b18 P 18892f6d43244679a593d7aad5877a09: Neither blocks nor log segments found. Creating new log.
I20250903 08:07:56.251130 15389 raft_consensus.cc:357] T c524a3070fae4d03b2328a218fb73b18 P c35d28b86cb846d399e9beb0a914d1d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe0343417e4407cacb543879bd8a7e9" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 39419 } } peers { permanent_uuid: "18892f6d43244679a593d7aad5877a09" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 45871 } } peers { permanent_uuid: "c35d28b86cb846d399e9beb0a914d1d4" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 41255 } }
I20250903 08:07:56.251250 15389 raft_consensus.cc:383] T c524a3070fae4d03b2328a218fb73b18 P c35d28b86cb846d399e9beb0a914d1d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:07:56.251295 15389 raft_consensus.cc:738] T c524a3070fae4d03b2328a218fb73b18 P c35d28b86cb846d399e9beb0a914d1d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c35d28b86cb846d399e9beb0a914d1d4, State: Initialized, Role: FOLLOWER
I20250903 08:07:56.251433 15389 consensus_queue.cc:260] T c524a3070fae4d03b2328a218fb73b18 P c35d28b86cb846d399e9beb0a914d1d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe0343417e4407cacb543879bd8a7e9" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 39419 } } peers { permanent_uuid: "18892f6d43244679a593d7aad5877a09" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 45871 } } peers { permanent_uuid: "c35d28b86cb846d399e9beb0a914d1d4" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 41255 } }
I20250903 08:07:56.251669 15395 tablet_replica.cc:440] T c524a3070fae4d03b2328a218fb73b18 P c35d28b86cb846d399e9beb0a914d1d4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe0343417e4407cacb543879bd8a7e9" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 39419 } } peers { permanent_uuid: "18892f6d43244679a593d7aad5877a09" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 45871 } } peers { permanent_uuid: "c35d28b86cb846d399e9beb0a914d1d4" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 41255 } } }
I20250903 08:07:56.251678 15389 ts_tablet_manager.cc:1428] T c524a3070fae4d03b2328a218fb73b18 P c35d28b86cb846d399e9beb0a914d1d4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250903 08:07:56.251806 15395 tablet_replica.cc:443] T c524a3070fae4d03b2328a218fb73b18 P c35d28b86cb846d399e9beb0a914d1d4: This TxnStatusTablet replica's current role is: FOLLOWER
I20250903 08:07:56.252287 15388 tablet_bootstrap.cc:492] T c524a3070fae4d03b2328a218fb73b18 P 18892f6d43244679a593d7aad5877a09: No bootstrap required, opened a new log
I20250903 08:07:56.252368 15387 tablet_bootstrap.cc:492] T c524a3070fae4d03b2328a218fb73b18 P 4fe0343417e4407cacb543879bd8a7e9: No bootstrap required, opened a new log
I20250903 08:07:56.252388 15388 ts_tablet_manager.cc:1397] T c524a3070fae4d03b2328a218fb73b18 P 18892f6d43244679a593d7aad5877a09: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250903 08:07:56.252449 15387 ts_tablet_manager.cc:1397] T c524a3070fae4d03b2328a218fb73b18 P 4fe0343417e4407cacb543879bd8a7e9: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20250903 08:07:56.252825 15388 raft_consensus.cc:357] T c524a3070fae4d03b2328a218fb73b18 P 18892f6d43244679a593d7aad5877a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe0343417e4407cacb543879bd8a7e9" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 39419 } } peers { permanent_uuid: "18892f6d43244679a593d7aad5877a09" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 45871 } } peers { permanent_uuid: "c35d28b86cb846d399e9beb0a914d1d4" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 41255 } }
I20250903 08:07:56.252864 15387 raft_consensus.cc:357] T c524a3070fae4d03b2328a218fb73b18 P 4fe0343417e4407cacb543879bd8a7e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe0343417e4407cacb543879bd8a7e9" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 39419 } } peers { permanent_uuid: "18892f6d43244679a593d7aad5877a09" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 45871 } } peers { permanent_uuid: "c35d28b86cb846d399e9beb0a914d1d4" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 41255 } }
I20250903 08:07:56.252934 15388 raft_consensus.cc:383] T c524a3070fae4d03b2328a218fb73b18 P 18892f6d43244679a593d7aad5877a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:07:56.252949 15387 raft_consensus.cc:383] T c524a3070fae4d03b2328a218fb73b18 P 4fe0343417e4407cacb543879bd8a7e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:07:56.252980 15388 raft_consensus.cc:738] T c524a3070fae4d03b2328a218fb73b18 P 18892f6d43244679a593d7aad5877a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18892f6d43244679a593d7aad5877a09, State: Initialized, Role: FOLLOWER
I20250903 08:07:56.252985 15387 raft_consensus.cc:738] T c524a3070fae4d03b2328a218fb73b18 P 4fe0343417e4407cacb543879bd8a7e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fe0343417e4407cacb543879bd8a7e9, State: Initialized, Role: FOLLOWER
I20250903 08:07:56.253108 15388 consensus_queue.cc:260] T c524a3070fae4d03b2328a218fb73b18 P 18892f6d43244679a593d7aad5877a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe0343417e4407cacb543879bd8a7e9" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 39419 } } peers { permanent_uuid: "18892f6d43244679a593d7aad5877a09" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 45871 } } peers { permanent_uuid: "c35d28b86cb846d399e9beb0a914d1d4" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 41255 } }
I20250903 08:07:56.253114 15387 consensus_queue.cc:260] T c524a3070fae4d03b2328a218fb73b18 P 4fe0343417e4407cacb543879bd8a7e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe0343417e4407cacb543879bd8a7e9" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 39419 } } peers { permanent_uuid: "18892f6d43244679a593d7aad5877a09" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 45871 } } peers { permanent_uuid: "c35d28b86cb846d399e9beb0a914d1d4" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 41255 } }
I20250903 08:07:56.253355 15387 ts_tablet_manager.cc:1428] T c524a3070fae4d03b2328a218fb73b18 P 4fe0343417e4407cacb543879bd8a7e9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250903 08:07:56.253371 15388 ts_tablet_manager.cc:1428] T c524a3070fae4d03b2328a218fb73b18 P 18892f6d43244679a593d7aad5877a09: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250903 08:07:56.253365 15394 tablet_replica.cc:440] T c524a3070fae4d03b2328a218fb73b18 P 18892f6d43244679a593d7aad5877a09: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe0343417e4407cacb543879bd8a7e9" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 39419 } } peers { permanent_uuid: "18892f6d43244679a593d7aad5877a09" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 45871 } } peers { permanent_uuid: "c35d28b86cb846d399e9beb0a914d1d4" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 41255 } } }
I20250903 08:07:56.253525 15394 tablet_replica.cc:443] T c524a3070fae4d03b2328a218fb73b18 P 18892f6d43244679a593d7aad5877a09: This TxnStatusTablet replica's current role is: FOLLOWER
I20250903 08:07:56.253732 15393 tablet_replica.cc:440] T c524a3070fae4d03b2328a218fb73b18 P 4fe0343417e4407cacb543879bd8a7e9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe0343417e4407cacb543879bd8a7e9" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 39419 } } peers { permanent_uuid: "18892f6d43244679a593d7aad5877a09" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 45871 } } peers { permanent_uuid: "c35d28b86cb846d399e9beb0a914d1d4" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 41255 } } }
I20250903 08:07:56.253822 15393 tablet_replica.cc:443] T c524a3070fae4d03b2328a218fb73b18 P 4fe0343417e4407cacb543879bd8a7e9: This TxnStatusTablet replica's current role is: FOLLOWER
I20250903 08:07:56.549192 15395 consensus_queue.cc:1035] T 85e451aecb5a461daae1bd11253031da P c35d28b86cb846d399e9beb0a914d1d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18892f6d43244679a593d7aad5877a09" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 45871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 08:07:56.554729 15395 consensus_queue.cc:1035] T 85e451aecb5a461daae1bd11253031da P c35d28b86cb846d399e9beb0a914d1d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fe0343417e4407cacb543879bd8a7e9" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 39419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 08:07:56.603708 15394 raft_consensus.cc:491] T c524a3070fae4d03b2328a218fb73b18 P 18892f6d43244679a593d7aad5877a09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 08:07:56.603864 15394 raft_consensus.cc:513] T c524a3070fae4d03b2328a218fb73b18 P 18892f6d43244679a593d7aad5877a09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe0343417e4407cacb543879bd8a7e9" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 39419 } } peers { permanent_uuid: "18892f6d43244679a593d7aad5877a09" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 45871 } } peers { permanent_uuid: "c35d28b86cb846d399e9beb0a914d1d4" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 41255 } }
I20250903 08:07:56.604914 15394 leader_election.cc:290] T c524a3070fae4d03b2328a218fb73b18 P 18892f6d43244679a593d7aad5877a09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4fe0343417e4407cacb543879bd8a7e9 (127.12.201.193:39419), c35d28b86cb846d399e9beb0a914d1d4 (127.12.201.195:41255)
I20250903 08:07:56.608628 15414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:56.605158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.193:39419 (local address 127.12.201.194:44673)
0903 08:07:56.605411 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0903 08:07:56.605428 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0903 08:07:56.605543 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:07:56.605965 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:07:56.605968 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:07:56.605989 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 08:07:56.606279 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:56.606288 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:56.607310 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:07:56.607314 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:07:56.608167 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:56.608175 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:56.608260 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:56.608279 (+ 19us) client_negotiation.cc:769] Sending connection context
0903 08:07:56.608327 (+ 48us) client_negotiation.cc:240] Negotiation successful
0903 08:07:56.608426 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":84,"threads_started":1}
I20250903 08:07:56.609071 15382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:56.605268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.194:44673 (local address 127.12.201.193:39419)
0903 08:07:56.605415 (+ 147us) server_negotiation.cc:207] Beginning negotiation
0903 08:07:56.605420 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 08:07:56.605500 (+ 80us) server_negotiation.cc:408] Connection header received
0903 08:07:56.605695 (+ 195us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:07:56.605702 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:07:56.605762 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:07:56.605814 (+ 52us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 08:07:56.606405 (+ 591us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:56.607210 (+ 805us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:07:56.608317 (+ 1107us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:56.608764 (+ 447us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:56.608802 (+ 38us) server_negotiation.cc:1036] Waiting for connection context
0903 08:07:56.608856 (+ 54us) server_negotiation.cc:300] Negotiation successful
0903 08:07:56.608926 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20250903 08:07:56.609208 15415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:56.605535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.195:41255 (local address 127.12.201.194:36507)
0903 08:07:56.605838 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0903 08:07:56.605854 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0903 08:07:56.605947 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:07:56.606303 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:07:56.606306 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:07:56.606321 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 08:07:56.607023 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:56.607033 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:56.608035 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:07:56.608038 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:07:56.608832 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:56.608841 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:56.608948 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:56.608964 (+ 16us) client_negotiation.cc:769] Sending connection context
0903 08:07:56.609016 (+ 52us) client_negotiation.cc:240] Negotiation successful
0903 08:07:56.609067 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":59,"threads_started":1}
I20250903 08:07:56.609474 15034 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c524a3070fae4d03b2328a218fb73b18" candidate_uuid: "18892f6d43244679a593d7aad5877a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fe0343417e4407cacb543879bd8a7e9" is_pre_election: true
I20250903 08:07:56.609597 15034 raft_consensus.cc:2466] T c524a3070fae4d03b2328a218fb73b18 P 4fe0343417e4407cacb543879bd8a7e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18892f6d43244679a593d7aad5877a09 in term 0.
I20250903 08:07:56.609702 15416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:56.605706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.194:36507 (local address 127.12.201.195:41255)
0903 08:07:56.605954 (+ 248us) server_negotiation.cc:207] Beginning negotiation
0903 08:07:56.605958 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:07:56.605972 (+ 14us) server_negotiation.cc:408] Connection header received
0903 08:07:56.606030 (+ 58us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:07:56.606033 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:07:56.606089 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:07:56.606165 (+ 76us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 08:07:56.607146 (+ 981us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:56.607889 (+ 743us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:07:56.608966 (+ 1077us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:56.609427 (+ 461us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:56.609464 (+ 37us) server_negotiation.cc:1036] Waiting for connection context
0903 08:07:56.609517 (+ 53us) server_negotiation.cc:300] Negotiation successful
0903 08:07:56.609575 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":91,"threads_started":1}
I20250903 08:07:56.609936 15107 leader_election.cc:304] T c524a3070fae4d03b2328a218fb73b18 P 18892f6d43244679a593d7aad5877a09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18892f6d43244679a593d7aad5877a09, 4fe0343417e4407cacb543879bd8a7e9; no voters:
I20250903 08:07:56.610210 15394 raft_consensus.cc:2802] T c524a3070fae4d03b2328a218fb73b18 P 18892f6d43244679a593d7aad5877a09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 08:07:56.610280 15394 raft_consensus.cc:491] T c524a3070fae4d03b2328a218fb73b18 P 18892f6d43244679a593d7aad5877a09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 08:07:56.610280 15318 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c524a3070fae4d03b2328a218fb73b18" candidate_uuid: "18892f6d43244679a593d7aad5877a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c35d28b86cb846d399e9beb0a914d1d4" is_pre_election: true
I20250903 08:07:56.610324 15394 raft_consensus.cc:3058] T c524a3070fae4d03b2328a218fb73b18 P 18892f6d43244679a593d7aad5877a09 [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:07:56.610472 15318 raft_consensus.cc:2466] T c524a3070fae4d03b2328a218fb73b18 P c35d28b86cb846d399e9beb0a914d1d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18892f6d43244679a593d7aad5877a09 in term 0.
I20250903 08:07:56.611311 15394 raft_consensus.cc:513] T c524a3070fae4d03b2328a218fb73b18 P 18892f6d43244679a593d7aad5877a09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe0343417e4407cacb543879bd8a7e9" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 39419 } } peers { permanent_uuid: "18892f6d43244679a593d7aad5877a09" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 45871 } } peers { permanent_uuid: "c35d28b86cb846d399e9beb0a914d1d4" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 41255 } }
I20250903 08:07:56.611642 15394 leader_election.cc:290] T c524a3070fae4d03b2328a218fb73b18 P 18892f6d43244679a593d7aad5877a09 [CANDIDATE]: Term 1 election: Requested vote from peers 4fe0343417e4407cacb543879bd8a7e9 (127.12.201.193:39419), c35d28b86cb846d399e9beb0a914d1d4 (127.12.201.195:41255)
I20250903 08:07:56.611984 15034 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c524a3070fae4d03b2328a218fb73b18" candidate_uuid: "18892f6d43244679a593d7aad5877a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fe0343417e4407cacb543879bd8a7e9"
I20250903 08:07:56.612030 15318 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c524a3070fae4d03b2328a218fb73b18" candidate_uuid: "18892f6d43244679a593d7aad5877a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c35d28b86cb846d399e9beb0a914d1d4"
I20250903 08:07:56.612104 15034 raft_consensus.cc:3058] T c524a3070fae4d03b2328a218fb73b18 P 4fe0343417e4407cacb543879bd8a7e9 [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:07:56.612150 15318 raft_consensus.cc:3058] T c524a3070fae4d03b2328a218fb73b18 P c35d28b86cb846d399e9beb0a914d1d4 [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:07:56.612893 15034 raft_consensus.cc:2466] T c524a3070fae4d03b2328a218fb73b18 P 4fe0343417e4407cacb543879bd8a7e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18892f6d43244679a593d7aad5877a09 in term 1.
I20250903 08:07:56.613060 15318 raft_consensus.cc:2466] T c524a3070fae4d03b2328a218fb73b18 P c35d28b86cb846d399e9beb0a914d1d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18892f6d43244679a593d7aad5877a09 in term 1.
I20250903 08:07:56.613202 15107 leader_election.cc:304] T c524a3070fae4d03b2328a218fb73b18 P 18892f6d43244679a593d7aad5877a09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18892f6d43244679a593d7aad5877a09, 4fe0343417e4407cacb543879bd8a7e9; no voters:
I20250903 08:07:56.613445 15394 raft_consensus.cc:2802] T c524a3070fae4d03b2328a218fb73b18 P 18892f6d43244679a593d7aad5877a09 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 08:07:56.613634 15394 raft_consensus.cc:695] T c524a3070fae4d03b2328a218fb73b18 P 18892f6d43244679a593d7aad5877a09 [term 1 LEADER]: Becoming Leader. State: Replica: 18892f6d43244679a593d7aad5877a09, State: Running, Role: LEADER
I20250903 08:07:56.613907 15394 consensus_queue.cc:237] T c524a3070fae4d03b2328a218fb73b18 P 18892f6d43244679a593d7aad5877a09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe0343417e4407cacb543879bd8a7e9" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 39419 } } peers { permanent_uuid: "18892f6d43244679a593d7aad5877a09" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 45871 } } peers { permanent_uuid: "c35d28b86cb846d399e9beb0a914d1d4" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 41255 } }
I20250903 08:07:56.615095 15417 tablet_replica.cc:440] T c524a3070fae4d03b2328a218fb73b18 P 18892f6d43244679a593d7aad5877a09: TxnStatusTablet state changed. Reason: New leader 18892f6d43244679a593d7aad5877a09. Latest consensus state: current_term: 1 leader_uuid: "18892f6d43244679a593d7aad5877a09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe0343417e4407cacb543879bd8a7e9" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 39419 } } peers { permanent_uuid: "18892f6d43244679a593d7aad5877a09" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 45871 } } peers { permanent_uuid: "c35d28b86cb846d399e9beb0a914d1d4" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 41255 } } }
I20250903 08:07:56.615212 15417 tablet_replica.cc:443] T c524a3070fae4d03b2328a218fb73b18 P 18892f6d43244679a593d7aad5877a09: This TxnStatusTablet replica's current role is: LEADER
I20250903 08:07:56.615473 15419 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20250903 08:07:56.615609 15419 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20250903 08:07:56.616199 14873 catalog_manager.cc:5582] T c524a3070fae4d03b2328a218fb73b18 P 18892f6d43244679a593d7aad5877a09 reported cstate change: term changed from 0 to 1, leader changed from <none> to 18892f6d43244679a593d7aad5877a09 (127.12.201.194). New cstate: current_term: 1 leader_uuid: "18892f6d43244679a593d7aad5877a09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe0343417e4407cacb543879bd8a7e9" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 39419 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18892f6d43244679a593d7aad5877a09" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 45871 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c35d28b86cb846d399e9beb0a914d1d4" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 41255 } health_report { overall_health: UNKNOWN } } }
I20250903 08:07:56.703984 15404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:56.700591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.194:45871 (local address 127.0.0.1:51964)
0903 08:07:56.700773 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0903 08:07:56.700791 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0903 08:07:56.700920 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:07:56.701271 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:07:56.701278 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:07:56.701290 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0903 08:07:56.701497 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:56.701503 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:56.702251 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:07:56.702255 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:07:56.702673 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:56.702680 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:56.702785 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:56.702833 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0903 08:07:56.703681 (+ 848us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0903 08:07:56.703697 (+ 16us) client_negotiation.cc:769] Sending connection context
0903 08:07:56.703771 (+ 74us) client_negotiation.cc:240] Negotiation successful
0903 08:07:56.703855 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20250903 08:07:56.703998 15422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:56.700710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51964 (local address 127.12.201.194:45871)
0903 08:07:56.700961 (+ 251us) server_negotiation.cc:207] Beginning negotiation
0903 08:07:56.700968 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0903 08:07:56.700987 (+ 19us) server_negotiation.cc:408] Connection header received
0903 08:07:56.701089 (+ 102us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:07:56.701091 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:07:56.701149 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:07:56.701242 (+ 93us) server_negotiation.cc:227] Negotiated authn=TOKEN
0903 08:07:56.701617 (+ 375us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:56.702135 (+ 518us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:07:56.702784 (+ 649us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:56.703032 (+ 248us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:56.703100 (+ 68us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0903 08:07:56.703576 (+ 476us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0903 08:07:56.703692 (+ 116us) server_negotiation.cc:1036] Waiting for connection context
0903 08:07:56.703788 (+ 96us) server_negotiation.cc:300] Negotiation successful
0903 08:07:56.703838 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":62,"threads_started":1}
I20250903 08:07:56.708879 15034 raft_consensus.cc:1273] T c524a3070fae4d03b2328a218fb73b18 P 4fe0343417e4407cacb543879bd8a7e9 [term 1 FOLLOWER]: Refusing update from remote peer 18892f6d43244679a593d7aad5877a09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 08:07:56.708956 15318 raft_consensus.cc:1273] T c524a3070fae4d03b2328a218fb73b18 P c35d28b86cb846d399e9beb0a914d1d4 [term 1 FOLLOWER]: Refusing update from remote peer 18892f6d43244679a593d7aad5877a09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 08:07:56.709460 15417 consensus_queue.cc:1035] T c524a3070fae4d03b2328a218fb73b18 P 18892f6d43244679a593d7aad5877a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fe0343417e4407cacb543879bd8a7e9" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 39419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 08:07:56.709609 15394 consensus_queue.cc:1035] T c524a3070fae4d03b2328a218fb73b18 P 18892f6d43244679a593d7aad5877a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c35d28b86cb846d399e9beb0a914d1d4" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 41255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 08:07:56.713420 15395 tablet_replica.cc:440] T c524a3070fae4d03b2328a218fb73b18 P c35d28b86cb846d399e9beb0a914d1d4: TxnStatusTablet state changed. Reason: New leader 18892f6d43244679a593d7aad5877a09. Latest consensus state: current_term: 1 leader_uuid: "18892f6d43244679a593d7aad5877a09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe0343417e4407cacb543879bd8a7e9" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 39419 } } peers { permanent_uuid: "18892f6d43244679a593d7aad5877a09" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 45871 } } peers { permanent_uuid: "c35d28b86cb846d399e9beb0a914d1d4" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 41255 } } }
I20250903 08:07:56.713529 15395 tablet_replica.cc:443] T c524a3070fae4d03b2328a218fb73b18 P c35d28b86cb846d399e9beb0a914d1d4: This TxnStatusTablet replica's current role is: FOLLOWER
I20250903 08:07:56.714250 15393 tablet_replica.cc:440] T c524a3070fae4d03b2328a218fb73b18 P 4fe0343417e4407cacb543879bd8a7e9: TxnStatusTablet state changed. Reason: New leader 18892f6d43244679a593d7aad5877a09. Latest consensus state: current_term: 1 leader_uuid: "18892f6d43244679a593d7aad5877a09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe0343417e4407cacb543879bd8a7e9" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 39419 } } peers { permanent_uuid: "18892f6d43244679a593d7aad5877a09" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 45871 } } peers { permanent_uuid: "c35d28b86cb846d399e9beb0a914d1d4" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 41255 } } }
I20250903 08:07:56.714357 15393 tablet_replica.cc:443] T c524a3070fae4d03b2328a218fb73b18 P 4fe0343417e4407cacb543879bd8a7e9: This TxnStatusTablet replica's current role is: FOLLOWER
I20250903 08:07:56.715515 15394 tablet_replica.cc:440] T c524a3070fae4d03b2328a218fb73b18 P 18892f6d43244679a593d7aad5877a09: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "18892f6d43244679a593d7aad5877a09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe0343417e4407cacb543879bd8a7e9" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 39419 } } peers { permanent_uuid: "18892f6d43244679a593d7aad5877a09" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 45871 } } peers { permanent_uuid: "c35d28b86cb846d399e9beb0a914d1d4" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 41255 } } }
I20250903 08:07:56.715611 15394 tablet_replica.cc:443] T c524a3070fae4d03b2328a218fb73b18 P 18892f6d43244679a593d7aad5877a09: This TxnStatusTablet replica's current role is: LEADER
I20250903 08:07:56.715842 15411 mvcc.cc:204] Tried to move back new op lower bound from 7196208646993506304 to 7196208646613053440. Current Snapshot: MvccSnapshot[applied={T|T < 7196208646993506304}]
I20250903 08:07:56.717221 15395 tablet_replica.cc:440] T c524a3070fae4d03b2328a218fb73b18 P c35d28b86cb846d399e9beb0a914d1d4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "18892f6d43244679a593d7aad5877a09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe0343417e4407cacb543879bd8a7e9" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 39419 } } peers { permanent_uuid: "18892f6d43244679a593d7aad5877a09" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 45871 } } peers { permanent_uuid: "c35d28b86cb846d399e9beb0a914d1d4" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 41255 } } }
I20250903 08:07:56.717332 15395 tablet_replica.cc:443] T c524a3070fae4d03b2328a218fb73b18 P c35d28b86cb846d399e9beb0a914d1d4: This TxnStatusTablet replica's current role is: FOLLOWER
I20250903 08:07:56.717456 15409 mvcc.cc:204] Tried to move back new op lower bound from 7196208646993506304 to 7196208646613053440. Current Snapshot: MvccSnapshot[applied={T|T < 7196208646993506304}]
I20250903 08:07:56.717955 15394 tablet_replica.cc:440] T c524a3070fae4d03b2328a218fb73b18 P 18892f6d43244679a593d7aad5877a09: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "18892f6d43244679a593d7aad5877a09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe0343417e4407cacb543879bd8a7e9" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 39419 } } peers { permanent_uuid: "18892f6d43244679a593d7aad5877a09" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 45871 } } peers { permanent_uuid: "c35d28b86cb846d399e9beb0a914d1d4" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 41255 } } }
I20250903 08:07:56.718050 15394 tablet_replica.cc:443] T c524a3070fae4d03b2328a218fb73b18 P 18892f6d43244679a593d7aad5877a09: This TxnStatusTablet replica's current role is: LEADER
I20250903 08:07:56.718894 15393 tablet_replica.cc:440] T c524a3070fae4d03b2328a218fb73b18 P 4fe0343417e4407cacb543879bd8a7e9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "18892f6d43244679a593d7aad5877a09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe0343417e4407cacb543879bd8a7e9" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 39419 } } peers { permanent_uuid: "18892f6d43244679a593d7aad5877a09" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 45871 } } peers { permanent_uuid: "c35d28b86cb846d399e9beb0a914d1d4" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 41255 } } }
I20250903 08:07:56.719019 15393 tablet_replica.cc:443] T c524a3070fae4d03b2328a218fb73b18 P 4fe0343417e4407cacb543879bd8a7e9: This TxnStatusTablet replica's current role is: FOLLOWER
I20250903 08:07:56.720432 15413 mvcc.cc:204] Tried to move back new op lower bound from 7196208646993506304 to 7196208646613053440. Current Snapshot: MvccSnapshot[applied={T|T < 7196208646993506304}]
I20250903 08:07:56.726521 15416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:56.723041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60356 (local address 127.12.201.195:41255)
0903 08:07:56.723145 (+ 104us) server_negotiation.cc:207] Beginning negotiation
0903 08:07:56.723148 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 08:07:56.723160 (+ 12us) server_negotiation.cc:408] Connection header received
0903 08:07:56.723336 (+ 176us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:07:56.723339 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:07:56.723420 (+ 81us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:07:56.723512 (+ 92us) server_negotiation.cc:227] Negotiated authn=TOKEN
0903 08:07:56.723984 (+ 472us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:56.724720 (+ 736us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:07:56.725610 (+ 890us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:56.725753 (+ 143us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:56.725799 (+ 46us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0903 08:07:56.726173 (+ 374us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0903 08:07:56.726261 (+ 88us) server_negotiation.cc:1036] Waiting for connection context
0903 08:07:56.726381 (+ 120us) server_negotiation.cc:300] Negotiation successful
0903 08:07:56.726423 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20250903 08:07:56.736825 15433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:56.733655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.194:45871 (local address 127.12.201.195:45779)
0903 08:07:56.734027 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0903 08:07:56.734039 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0903 08:07:56.734129 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:07:56.734403 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:07:56.734406 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:07:56.734412 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0903 08:07:56.734610 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:56.734614 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:56.735620 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:07:56.735623 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:07:56.736021 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:56.736028 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:56.736121 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:56.736153 (+ 32us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0903 08:07:56.736597 (+ 444us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0903 08:07:56.736601 (+ 4us) client_negotiation.cc:769] Sending connection context
0903 08:07:56.736656 (+ 55us) client_negotiation.cc:240] Negotiation successful
0903 08:07:56.736711 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":103,"threads_started":1}
I20250903 08:07:56.736860 15422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:56.733773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.195:45779 (local address 127.12.201.194:45871)
0903 08:07:56.733889 (+ 116us) server_negotiation.cc:207] Beginning negotiation
0903 08:07:56.733896 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0903 08:07:56.734116 (+ 220us) server_negotiation.cc:408] Connection header received
0903 08:07:56.734216 (+ 100us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:07:56.734220 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:07:56.734290 (+ 70us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:07:56.734393 (+ 103us) server_negotiation.cc:227] Negotiated authn=TOKEN
0903 08:07:56.734715 (+ 322us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:56.735513 (+ 798us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:07:56.736136 (+ 623us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:56.736371 (+ 235us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:56.736421 (+ 50us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0903 08:07:56.736503 (+ 82us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0903 08:07:56.736579 (+ 76us) server_negotiation.cc:1036] Waiting for connection context
0903 08:07:56.736680 (+ 101us) server_negotiation.cc:300] Negotiation successful
0903 08:07:56.736731 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20250903 08:07:56.762233 15435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:56.758424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.195:41255 (local address 127.12.201.194:41895)
0903 08:07:56.759108 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0903 08:07:56.759122 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0903 08:07:56.759231 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:07:56.759634 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:07:56.759637 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:07:56.759646 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0903 08:07:56.759948 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:56.759957 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:56.760943 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:07:56.760945 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:07:56.761384 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:56.761391 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:56.761504 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:56.761551 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0903 08:07:56.761935 (+ 384us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0903 08:07:56.761942 (+ 7us) client_negotiation.cc:769] Sending connection context
0903 08:07:56.762017 (+ 75us) client_negotiation.cc:240] Negotiation successful
0903 08:07:56.762105 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":68,"threads_started":1}
I20250903 08:07:56.762228 15416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:56.758475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.194:41895 (local address 127.12.201.195:41255)
0903 08:07:56.758612 (+ 137us) server_negotiation.cc:207] Beginning negotiation
0903 08:07:56.758616 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:07:56.759458 (+ 842us) server_negotiation.cc:408] Connection header received
0903 08:07:56.759498 (+ 40us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:07:56.759501 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:07:56.759543 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:07:56.759613 (+ 70us) server_negotiation.cc:227] Negotiated authn=TOKEN
0903 08:07:56.760127 (+ 514us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:56.760838 (+ 711us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:07:56.761501 (+ 663us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:56.761696 (+ 195us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:56.761750 (+ 54us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0903 08:07:56.761838 (+ 88us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0903 08:07:56.761922 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
0903 08:07:56.762041 (+ 119us) server_negotiation.cc:300] Negotiation successful
0903 08:07:56.762091 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20250903 08:07:56.787947 14873 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53140:
table { table_name: "test-table" } modify_external_catalogs: true
I20250903 08:07:56.788156 14873 catalog_manager.cc:2730] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53140:
table { table_name: "test-table" } modify_external_catalogs: true
I20250903 08:07:56.790758 14873 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 911c1d9d6e52403ca64f3910291fda10: Sending DeleteTablet for 3 replicas of tablet 85e451aecb5a461daae1bd11253031da
I20250903 08:07:56.791946 15014 tablet_service.cc:1515] Processing DeleteTablet for tablet 85e451aecb5a461daae1bd11253031da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 08:07:56 UTC) from {username='kudu', principal='kudu/127.12.201.254@KRBTEST.COM'} at 127.0.0.1:51148
I20250903 08:07:56.792032 14873 catalog_manager.cc:2232] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53140:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 08:07:56.792418 14873 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 08:07:56.792415 15298 tablet_service.cc:1515] Processing DeleteTablet for tablet 85e451aecb5a461daae1bd11253031da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 08:07:56 UTC) from {username='kudu', principal='kudu/127.12.201.254@KRBTEST.COM'} at 127.0.0.1:60350
I20250903 08:07:56.793077 15156 tablet_service.cc:1515] Processing DeleteTablet for tablet 85e451aecb5a461daae1bd11253031da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 08:07:56 UTC) from {username='kudu', principal='kudu/127.12.201.254@KRBTEST.COM'} at 127.0.0.1:51956
I20250903 08:07:56.793588 15438 tablet_replica.cc:331] T 85e451aecb5a461daae1bd11253031da P 4fe0343417e4407cacb543879bd8a7e9: stopping tablet replica
I20250903 08:07:56.793790 15439 tablet_replica.cc:331] T 85e451aecb5a461daae1bd11253031da P c35d28b86cb846d399e9beb0a914d1d4: stopping tablet replica
I20250903 08:07:56.793984 15440 tablet_replica.cc:331] T 85e451aecb5a461daae1bd11253031da P 18892f6d43244679a593d7aad5877a09: stopping tablet replica
I20250903 08:07:56.794317 15440 raft_consensus.cc:2241] T 85e451aecb5a461daae1bd11253031da P 18892f6d43244679a593d7aad5877a09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 08:07:56.794588 15439 raft_consensus.cc:2241] T 85e451aecb5a461daae1bd11253031da P c35d28b86cb846d399e9beb0a914d1d4 [term 1 LEADER]: Raft consensus shutting down.
I20250903 08:07:56.794867 15438 raft_consensus.cc:2241] T 85e451aecb5a461daae1bd11253031da P 4fe0343417e4407cacb543879bd8a7e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 08:07:56.795289 15439 raft_consensus.cc:2270] T 85e451aecb5a461daae1bd11253031da P c35d28b86cb846d399e9beb0a914d1d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 08:07:56.795550 15438 raft_consensus.cc:2270] T 85e451aecb5a461daae1bd11253031da P 4fe0343417e4407cacb543879bd8a7e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 08:07:56.795742 15440 raft_consensus.cc:2270] T 85e451aecb5a461daae1bd11253031da P 18892f6d43244679a593d7aad5877a09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 08:07:56.797322 15014 tablet_service.cc:1468] Processing CreateTablet for tablet 85020976e2fe4523bec20360a7b56599 (DEFAULT_TABLE table=test-table [id=8f3846e7cff04b8ea18185526271d068]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:07:56.797617 15014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85020976e2fe4523bec20360a7b56599. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:07:56.798558 15298 tablet_service.cc:1468] Processing CreateTablet for tablet 85020976e2fe4523bec20360a7b56599 (DEFAULT_TABLE table=test-table [id=8f3846e7cff04b8ea18185526271d068]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:07:56.798836 15298 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85020976e2fe4523bec20360a7b56599. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:07:56.799420 15438 ts_tablet_manager.cc:1905] T 85e451aecb5a461daae1bd11253031da P 4fe0343417e4407cacb543879bd8a7e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 08:07:56.799640 15439 ts_tablet_manager.cc:1905] T 85e451aecb5a461daae1bd11253031da P c35d28b86cb846d399e9beb0a914d1d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 08:07:56.800387 15441 tablet_bootstrap.cc:492] T 85020976e2fe4523bec20360a7b56599 P 4fe0343417e4407cacb543879bd8a7e9: Bootstrap starting.
I20250903 08:07:56.797405 15440 ts_tablet_manager.cc:1905] T 85e451aecb5a461daae1bd11253031da P 18892f6d43244679a593d7aad5877a09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 08:07:56.801358 15156 tablet_service.cc:1468] Processing CreateTablet for tablet 85020976e2fe4523bec20360a7b56599 (DEFAULT_TABLE table=test-table [id=8f3846e7cff04b8ea18185526271d068]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:07:56.801980 15156 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85020976e2fe4523bec20360a7b56599. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:07:56.803061 15441 tablet_bootstrap.cc:654] T 85020976e2fe4523bec20360a7b56599 P 4fe0343417e4407cacb543879bd8a7e9: Neither blocks nor log segments found. Creating new log.
I20250903 08:07:56.803380 15439 ts_tablet_manager.cc:1918] T 85e451aecb5a461daae1bd11253031da P c35d28b86cb846d399e9beb0a914d1d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20250903 08:07:56.803507 15439 log.cc:1199] T 85e451aecb5a461daae1bd11253031da P c35d28b86cb846d399e9beb0a914d1d4: Deleting WAL directory at /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/minicluster-data/ts-2/wal/wals/85e451aecb5a461daae1bd11253031da
I20250903 08:07:56.803893 15439 ts_tablet_manager.cc:1939] T 85e451aecb5a461daae1bd11253031da P c35d28b86cb846d399e9beb0a914d1d4: Deleting consensus metadata
I20250903 08:07:56.805008 14859 catalog_manager.cc:4928] TS c35d28b86cb846d399e9beb0a914d1d4 (127.12.201.195:41255): tablet 85e451aecb5a461daae1bd11253031da (table test-table [id=9eac45e9bea44060bce09117a2dfcdce]) successfully deleted
I20250903 08:07:56.805423 15444 tablet_bootstrap.cc:492] T 85020976e2fe4523bec20360a7b56599 P 18892f6d43244679a593d7aad5877a09: Bootstrap starting.
I20250903 08:07:56.806358 15444 tablet_bootstrap.cc:654] T 85020976e2fe4523bec20360a7b56599 P 18892f6d43244679a593d7aad5877a09: Neither blocks nor log segments found. Creating new log.
I20250903 08:07:56.806668 15440 ts_tablet_manager.cc:1918] T 85e451aecb5a461daae1bd11253031da P 18892f6d43244679a593d7aad5877a09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20250903 08:07:56.806761 15440 log.cc:1199] T 85e451aecb5a461daae1bd11253031da P 18892f6d43244679a593d7aad5877a09: Deleting WAL directory at /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/minicluster-data/ts-1/wal/wals/85e451aecb5a461daae1bd11253031da
I20250903 08:07:56.807091 15440 ts_tablet_manager.cc:1939] T 85e451aecb5a461daae1bd11253031da P 18892f6d43244679a593d7aad5877a09: Deleting consensus metadata
I20250903 08:07:56.807523 15438 ts_tablet_manager.cc:1918] T 85e451aecb5a461daae1bd11253031da P 4fe0343417e4407cacb543879bd8a7e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20250903 08:07:56.807602 15438 log.cc:1199] T 85e451aecb5a461daae1bd11253031da P 4fe0343417e4407cacb543879bd8a7e9: Deleting WAL directory at /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/minicluster-data/ts-0/wal/wals/85e451aecb5a461daae1bd11253031da
I20250903 08:07:56.807703 15444 tablet_bootstrap.cc:492] T 85020976e2fe4523bec20360a7b56599 P 18892f6d43244679a593d7aad5877a09: No bootstrap required, opened a new log
I20250903 08:07:56.807777 15444 ts_tablet_manager.cc:1397] T 85020976e2fe4523bec20360a7b56599 P 18892f6d43244679a593d7aad5877a09: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250903 08:07:56.807910 15438 ts_tablet_manager.cc:1939] T 85e451aecb5a461daae1bd11253031da P 4fe0343417e4407cacb543879bd8a7e9: Deleting consensus metadata
I20250903 08:07:56.808182 15444 raft_consensus.cc:357] T 85020976e2fe4523bec20360a7b56599 P 18892f6d43244679a593d7aad5877a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe0343417e4407cacb543879bd8a7e9" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 39419 } } peers { permanent_uuid: "18892f6d43244679a593d7aad5877a09" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 45871 } } peers { permanent_uuid: "c35d28b86cb846d399e9beb0a914d1d4" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 41255 } }
I20250903 08:07:56.808247 15442 tablet_bootstrap.cc:492] T 85020976e2fe4523bec20360a7b56599 P c35d28b86cb846d399e9beb0a914d1d4: Bootstrap starting.
I20250903 08:07:56.808277 15444 raft_consensus.cc:383] T 85020976e2fe4523bec20360a7b56599 P 18892f6d43244679a593d7aad5877a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:07:56.808317 15444 raft_consensus.cc:738] T 85020976e2fe4523bec20360a7b56599 P 18892f6d43244679a593d7aad5877a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18892f6d43244679a593d7aad5877a09, State: Initialized, Role: FOLLOWER
I20250903 08:07:56.808434 15444 consensus_queue.cc:260] T 85020976e2fe4523bec20360a7b56599 P 18892f6d43244679a593d7aad5877a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe0343417e4407cacb543879bd8a7e9" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 39419 } } peers { permanent_uuid: "18892f6d43244679a593d7aad5877a09" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 45871 } } peers { permanent_uuid: "c35d28b86cb846d399e9beb0a914d1d4" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 41255 } }
I20250903 08:07:56.808671 15444 ts_tablet_manager.cc:1428] T 85020976e2fe4523bec20360a7b56599 P 18892f6d43244679a593d7aad5877a09: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250903 08:07:56.808998 15441 tablet_bootstrap.cc:492] T 85020976e2fe4523bec20360a7b56599 P 4fe0343417e4407cacb543879bd8a7e9: No bootstrap required, opened a new log
I20250903 08:07:56.809082 15441 ts_tablet_manager.cc:1397] T 85020976e2fe4523bec20360a7b56599 P 4fe0343417e4407cacb543879bd8a7e9: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20250903 08:07:56.809602 15441 raft_consensus.cc:357] T 85020976e2fe4523bec20360a7b56599 P 4fe0343417e4407cacb543879bd8a7e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe0343417e4407cacb543879bd8a7e9" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 39419 } } peers { permanent_uuid: "18892f6d43244679a593d7aad5877a09" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 45871 } } peers { permanent_uuid: "c35d28b86cb846d399e9beb0a914d1d4" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 41255 } }
I20250903 08:07:56.809711 15441 raft_consensus.cc:383] T 85020976e2fe4523bec20360a7b56599 P 4fe0343417e4407cacb543879bd8a7e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:07:56.809748 15441 raft_consensus.cc:738] T 85020976e2fe4523bec20360a7b56599 P 4fe0343417e4407cacb543879bd8a7e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fe0343417e4407cacb543879bd8a7e9, State: Initialized, Role: FOLLOWER
I20250903 08:07:56.809875 15441 consensus_queue.cc:260] T 85020976e2fe4523bec20360a7b56599 P 4fe0343417e4407cacb543879bd8a7e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe0343417e4407cacb543879bd8a7e9" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 39419 } } peers { permanent_uuid: "18892f6d43244679a593d7aad5877a09" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 45871 } } peers { permanent_uuid: "c35d28b86cb846d399e9beb0a914d1d4" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 41255 } }
I20250903 08:07:56.810148 15442 tablet_bootstrap.cc:654] T 85020976e2fe4523bec20360a7b56599 P c35d28b86cb846d399e9beb0a914d1d4: Neither blocks nor log segments found. Creating new log.
I20250903 08:07:56.810168 15441 ts_tablet_manager.cc:1428] T 85020976e2fe4523bec20360a7b56599 P 4fe0343417e4407cacb543879bd8a7e9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250903 08:07:56.812285 14859 catalog_manager.cc:4928] TS 4fe0343417e4407cacb543879bd8a7e9 (127.12.201.193:39419): tablet 85e451aecb5a461daae1bd11253031da (table test-table [id=9eac45e9bea44060bce09117a2dfcdce]) successfully deleted
I20250903 08:07:56.812655 14859 catalog_manager.cc:4928] TS 18892f6d43244679a593d7aad5877a09 (127.12.201.194:45871): tablet 85e451aecb5a461daae1bd11253031da (table test-table [id=9eac45e9bea44060bce09117a2dfcdce]) successfully deleted
I20250903 08:07:56.813580 15442 tablet_bootstrap.cc:492] T 85020976e2fe4523bec20360a7b56599 P c35d28b86cb846d399e9beb0a914d1d4: No bootstrap required, opened a new log
I20250903 08:07:56.813709 15442 ts_tablet_manager.cc:1397] T 85020976e2fe4523bec20360a7b56599 P c35d28b86cb846d399e9beb0a914d1d4: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20250903 08:07:56.814203 15442 raft_consensus.cc:357] T 85020976e2fe4523bec20360a7b56599 P c35d28b86cb846d399e9beb0a914d1d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe0343417e4407cacb543879bd8a7e9" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 39419 } } peers { permanent_uuid: "18892f6d43244679a593d7aad5877a09" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 45871 } } peers { permanent_uuid: "c35d28b86cb846d399e9beb0a914d1d4" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 41255 } }
I20250903 08:07:56.814370 15442 raft_consensus.cc:383] T 85020976e2fe4523bec20360a7b56599 P c35d28b86cb846d399e9beb0a914d1d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:07:56.814443 15442 raft_consensus.cc:738] T 85020976e2fe4523bec20360a7b56599 P c35d28b86cb846d399e9beb0a914d1d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c35d28b86cb846d399e9beb0a914d1d4, State: Initialized, Role: FOLLOWER
I20250903 08:07:56.814610 15442 consensus_queue.cc:260] T 85020976e2fe4523bec20360a7b56599 P c35d28b86cb846d399e9beb0a914d1d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe0343417e4407cacb543879bd8a7e9" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 39419 } } peers { permanent_uuid: "18892f6d43244679a593d7aad5877a09" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 45871 } } peers { permanent_uuid: "c35d28b86cb846d399e9beb0a914d1d4" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 41255 } }
I20250903 08:07:56.814924 15442 ts_tablet_manager.cc:1428] T 85020976e2fe4523bec20360a7b56599 P c35d28b86cb846d399e9beb0a914d1d4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250903 08:07:57.094427 15425 raft_consensus.cc:491] T 85020976e2fe4523bec20360a7b56599 P 18892f6d43244679a593d7aad5877a09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 08:07:57.094579 15425 raft_consensus.cc:513] T 85020976e2fe4523bec20360a7b56599 P 18892f6d43244679a593d7aad5877a09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe0343417e4407cacb543879bd8a7e9" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 39419 } } peers { permanent_uuid: "18892f6d43244679a593d7aad5877a09" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 45871 } } peers { permanent_uuid: "c35d28b86cb846d399e9beb0a914d1d4" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 41255 } }
I20250903 08:07:57.095069 15425 leader_election.cc:290] T 85020976e2fe4523bec20360a7b56599 P 18892f6d43244679a593d7aad5877a09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4fe0343417e4407cacb543879bd8a7e9 (127.12.201.193:39419), c35d28b86cb846d399e9beb0a914d1d4 (127.12.201.195:41255)
I20250903 08:07:57.095577 15318 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "85020976e2fe4523bec20360a7b56599" candidate_uuid: "18892f6d43244679a593d7aad5877a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c35d28b86cb846d399e9beb0a914d1d4" is_pre_election: true
I20250903 08:07:57.095587 15034 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "85020976e2fe4523bec20360a7b56599" candidate_uuid: "18892f6d43244679a593d7aad5877a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fe0343417e4407cacb543879bd8a7e9" is_pre_election: true
I20250903 08:07:57.095715 15318 raft_consensus.cc:2466] T 85020976e2fe4523bec20360a7b56599 P c35d28b86cb846d399e9beb0a914d1d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18892f6d43244679a593d7aad5877a09 in term 0.
I20250903 08:07:57.095733 15034 raft_consensus.cc:2466] T 85020976e2fe4523bec20360a7b56599 P 4fe0343417e4407cacb543879bd8a7e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18892f6d43244679a593d7aad5877a09 in term 0.
I20250903 08:07:57.096066 15107 leader_election.cc:304] T 85020976e2fe4523bec20360a7b56599 P 18892f6d43244679a593d7aad5877a09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18892f6d43244679a593d7aad5877a09, c35d28b86cb846d399e9beb0a914d1d4; no voters:
I20250903 08:07:57.096292 15425 raft_consensus.cc:2802] T 85020976e2fe4523bec20360a7b56599 P 18892f6d43244679a593d7aad5877a09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 08:07:57.096351 15425 raft_consensus.cc:491] T 85020976e2fe4523bec20360a7b56599 P 18892f6d43244679a593d7aad5877a09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 08:07:57.096387 15425 raft_consensus.cc:3058] T 85020976e2fe4523bec20360a7b56599 P 18892f6d43244679a593d7aad5877a09 [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:07:57.097530 15425 raft_consensus.cc:513] T 85020976e2fe4523bec20360a7b56599 P 18892f6d43244679a593d7aad5877a09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe0343417e4407cacb543879bd8a7e9" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 39419 } } peers { permanent_uuid: "18892f6d43244679a593d7aad5877a09" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 45871 } } peers { permanent_uuid: "c35d28b86cb846d399e9beb0a914d1d4" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 41255 } }
I20250903 08:07:57.097906 15425 leader_election.cc:290] T 85020976e2fe4523bec20360a7b56599 P 18892f6d43244679a593d7aad5877a09 [CANDIDATE]: Term 1 election: Requested vote from peers 4fe0343417e4407cacb543879bd8a7e9 (127.12.201.193:39419), c35d28b86cb846d399e9beb0a914d1d4 (127.12.201.195:41255)
I20250903 08:07:57.098649 15034 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "85020976e2fe4523bec20360a7b56599" candidate_uuid: "18892f6d43244679a593d7aad5877a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fe0343417e4407cacb543879bd8a7e9"
I20250903 08:07:57.098652 15318 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "85020976e2fe4523bec20360a7b56599" candidate_uuid: "18892f6d43244679a593d7aad5877a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c35d28b86cb846d399e9beb0a914d1d4"
I20250903 08:07:57.098791 15034 raft_consensus.cc:3058] T 85020976e2fe4523bec20360a7b56599 P 4fe0343417e4407cacb543879bd8a7e9 [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:07:57.098791 15318 raft_consensus.cc:3058] T 85020976e2fe4523bec20360a7b56599 P c35d28b86cb846d399e9beb0a914d1d4 [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:07:57.099922 15318 raft_consensus.cc:2466] T 85020976e2fe4523bec20360a7b56599 P c35d28b86cb846d399e9beb0a914d1d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18892f6d43244679a593d7aad5877a09 in term 1.
I20250903 08:07:57.099922 15034 raft_consensus.cc:2466] T 85020976e2fe4523bec20360a7b56599 P 4fe0343417e4407cacb543879bd8a7e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18892f6d43244679a593d7aad5877a09 in term 1.
I20250903 08:07:57.100260 15107 leader_election.cc:304] T 85020976e2fe4523bec20360a7b56599 P 18892f6d43244679a593d7aad5877a09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18892f6d43244679a593d7aad5877a09, c35d28b86cb846d399e9beb0a914d1d4; no voters:
I20250903 08:07:57.100530 15425 raft_consensus.cc:2802] T 85020976e2fe4523bec20360a7b56599 P 18892f6d43244679a593d7aad5877a09 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 08:07:57.100605 15425 raft_consensus.cc:695] T 85020976e2fe4523bec20360a7b56599 P 18892f6d43244679a593d7aad5877a09 [term 1 LEADER]: Becoming Leader. State: Replica: 18892f6d43244679a593d7aad5877a09, State: Running, Role: LEADER
I20250903 08:07:57.100747 15425 consensus_queue.cc:237] T 85020976e2fe4523bec20360a7b56599 P 18892f6d43244679a593d7aad5877a09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe0343417e4407cacb543879bd8a7e9" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 39419 } } peers { permanent_uuid: "18892f6d43244679a593d7aad5877a09" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 45871 } } peers { permanent_uuid: "c35d28b86cb846d399e9beb0a914d1d4" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 41255 } }
I20250903 08:07:57.102473 14873 catalog_manager.cc:5582] T 85020976e2fe4523bec20360a7b56599 P 18892f6d43244679a593d7aad5877a09 reported cstate change: term changed from 0 to 1, leader changed from <none> to 18892f6d43244679a593d7aad5877a09 (127.12.201.194). New cstate: current_term: 1 leader_uuid: "18892f6d43244679a593d7aad5877a09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe0343417e4407cacb543879bd8a7e9" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 39419 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18892f6d43244679a593d7aad5877a09" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 45871 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c35d28b86cb846d399e9beb0a914d1d4" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 41255 } health_report { overall_health: UNKNOWN } } }
I20250903 08:07:57.178216 15422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:57.175847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51980 (local address 127.12.201.194:45871)
0903 08:07:57.175978 (+ 131us) server_negotiation.cc:207] Beginning negotiation
0903 08:07:57.175982 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:07:57.175996 (+ 14us) server_negotiation.cc:408] Connection header received
0903 08:07:57.176113 (+ 117us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:07:57.176115 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:07:57.176162 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:07:57.176225 (+ 63us) server_negotiation.cc:227] Negotiated authn=TOKEN
0903 08:07:57.176533 (+ 308us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:57.176981 (+ 448us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:07:57.177586 (+ 605us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:57.177754 (+ 168us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:57.177794 (+ 40us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0903 08:07:57.177860 (+ 66us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0903 08:07:57.177925 (+ 65us) server_negotiation.cc:1036] Waiting for connection context
0903 08:07:57.178038 (+ 113us) server_negotiation.cc:300] Negotiation successful
0903 08:07:57.178082 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20250903 08:07:57.185547 15435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:57.183181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.194:45871 (local address 127.12.201.194:50893)
0903 08:07:57.183304 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0903 08:07:57.183331 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0903 08:07:57.183425 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:07:57.183616 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:07:57.183619 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:07:57.183625 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0903 08:07:57.183780 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:57.183784 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:57.184444 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:07:57.184448 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:07:57.184820 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:57.184827 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:57.184928 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:57.184951 (+ 23us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0903 08:07:57.185263 (+ 312us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0903 08:07:57.185268 (+ 5us) client_negotiation.cc:769] Sending connection context
0903 08:07:57.185332 (+ 64us) client_negotiation.cc:240] Negotiation successful
0903 08:07:57.185400 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20250903 08:07:57.185547 15422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:57.183234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.194:50893 (local address 127.12.201.194:45871)
0903 08:07:57.183327 (+ 93us) server_negotiation.cc:207] Beginning negotiation
0903 08:07:57.183332 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 08:07:57.183387 (+ 55us) server_negotiation.cc:408] Connection header received
0903 08:07:57.183503 (+ 116us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:07:57.183506 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:07:57.183544 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:07:57.183606 (+ 62us) server_negotiation.cc:227] Negotiated authn=TOKEN
0903 08:07:57.183876 (+ 270us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:57.184347 (+ 471us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:07:57.184919 (+ 572us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:57.185093 (+ 174us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:57.185126 (+ 33us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0903 08:07:57.185179 (+ 53us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0903 08:07:57.185243 (+ 64us) server_negotiation.cc:1036] Waiting for connection context
0903 08:07:57.185357 (+ 114us) server_negotiation.cc:300] Negotiation successful
0903 08:07:57.185390 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20250903 08:07:57.190546 15318 raft_consensus.cc:1273] T 85020976e2fe4523bec20360a7b56599 P c35d28b86cb846d399e9beb0a914d1d4 [term 1 FOLLOWER]: Refusing update from remote peer 18892f6d43244679a593d7aad5877a09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 08:07:57.190588 15034 raft_consensus.cc:1273] T 85020976e2fe4523bec20360a7b56599 P 4fe0343417e4407cacb543879bd8a7e9 [term 1 FOLLOWER]: Refusing update from remote peer 18892f6d43244679a593d7aad5877a09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 08:07:57.190881 15425 consensus_queue.cc:1035] T 85020976e2fe4523bec20360a7b56599 P 18892f6d43244679a593d7aad5877a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c35d28b86cb846d399e9beb0a914d1d4" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 41255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 08:07:57.191028 15417 consensus_queue.cc:1035] T 85020976e2fe4523bec20360a7b56599 P 18892f6d43244679a593d7aad5877a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fe0343417e4407cacb543879bd8a7e9" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 39419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250903 08:07:57.203917 15411 tablet_replica.cc:1304] Aborted: operation has been aborted: cancelling pending write operations
I20250903 08:07:58.754534 14873 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53140:
table { table_name: "test-table" } modify_external_catalogs: true
I20250903 08:07:58.754679 14873 catalog_manager.cc:2730] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53140:
table { table_name: "test-table" } modify_external_catalogs: true
I20250903 08:07:58.756860 14873 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 911c1d9d6e52403ca64f3910291fda10: Sending DeleteTablet for 3 replicas of tablet 85020976e2fe4523bec20360a7b56599
I20250903 08:07:58.757398 15156 tablet_service.cc:1515] Processing DeleteTablet for tablet 85020976e2fe4523bec20360a7b56599 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 08:07:58 UTC) from {username='kudu', principal='kudu/127.12.201.254@KRBTEST.COM'} at 127.0.0.1:51956
I20250903 08:07:58.757472 15298 tablet_service.cc:1515] Processing DeleteTablet for tablet 85020976e2fe4523bec20360a7b56599 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 08:07:58 UTC) from {username='kudu', principal='kudu/127.12.201.254@KRBTEST.COM'} at 127.0.0.1:60350
I20250903 08:07:58.757472 15014 tablet_service.cc:1515] Processing DeleteTablet for tablet 85020976e2fe4523bec20360a7b56599 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 08:07:58 UTC) from {username='kudu', principal='kudu/127.12.201.254@KRBTEST.COM'} at 127.0.0.1:51148
I20250903 08:07:58.758222 15472 tablet_replica.cc:331] T 85020976e2fe4523bec20360a7b56599 P 4fe0343417e4407cacb543879bd8a7e9: stopping tablet replica
I20250903 08:07:58.758350 15472 raft_consensus.cc:2241] T 85020976e2fe4523bec20360a7b56599 P 4fe0343417e4407cacb543879bd8a7e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 08:07:58.758409 15471 tablet_replica.cc:331] T 85020976e2fe4523bec20360a7b56599 P 18892f6d43244679a593d7aad5877a09: stopping tablet replica
I20250903 08:07:58.758467 15472 raft_consensus.cc:2270] T 85020976e2fe4523bec20360a7b56599 P 4fe0343417e4407cacb543879bd8a7e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 08:07:58.758512 15471 raft_consensus.cc:2241] T 85020976e2fe4523bec20360a7b56599 P 18892f6d43244679a593d7aad5877a09 [term 1 LEADER]: Raft consensus shutting down.
I20250903 08:07:58.758662 15473 tablet_replica.cc:331] T 85020976e2fe4523bec20360a7b56599 P c35d28b86cb846d399e9beb0a914d1d4: stopping tablet replica
I20250903 08:07:58.758751 15473 raft_consensus.cc:2241] T 85020976e2fe4523bec20360a7b56599 P c35d28b86cb846d399e9beb0a914d1d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 08:07:58.758798 15471 raft_consensus.cc:2270] T 85020976e2fe4523bec20360a7b56599 P 18892f6d43244679a593d7aad5877a09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 08:07:58.758847 15473 raft_consensus.cc:2270] T 85020976e2fe4523bec20360a7b56599 P c35d28b86cb846d399e9beb0a914d1d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 08:07:58.759450 15473 ts_tablet_manager.cc:1905] T 85020976e2fe4523bec20360a7b56599 P c35d28b86cb846d399e9beb0a914d1d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 08:07:58.759450 15471 ts_tablet_manager.cc:1905] T 85020976e2fe4523bec20360a7b56599 P 18892f6d43244679a593d7aad5877a09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 08:07:58.760739 15472 ts_tablet_manager.cc:1905] T 85020976e2fe4523bec20360a7b56599 P 4fe0343417e4407cacb543879bd8a7e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 08:07:58.761952 15471 ts_tablet_manager.cc:1918] T 85020976e2fe4523bec20360a7b56599 P 18892f6d43244679a593d7aad5877a09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20250903 08:07:58.762032 15471 log.cc:1199] T 85020976e2fe4523bec20360a7b56599 P 18892f6d43244679a593d7aad5877a09: Deleting WAL directory at /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/minicluster-data/ts-1/wal/wals/85020976e2fe4523bec20360a7b56599
I20250903 08:07:58.762152 15473 ts_tablet_manager.cc:1918] T 85020976e2fe4523bec20360a7b56599 P c35d28b86cb846d399e9beb0a914d1d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20250903 08:07:58.762221 15473 log.cc:1199] T 85020976e2fe4523bec20360a7b56599 P c35d28b86cb846d399e9beb0a914d1d4: Deleting WAL directory at /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/minicluster-data/ts-2/wal/wals/85020976e2fe4523bec20360a7b56599
I20250903 08:07:58.762354 15471 ts_tablet_manager.cc:1939] T 85020976e2fe4523bec20360a7b56599 P 18892f6d43244679a593d7aad5877a09: Deleting consensus metadata
I20250903 08:07:58.762517 15473 ts_tablet_manager.cc:1939] T 85020976e2fe4523bec20360a7b56599 P c35d28b86cb846d399e9beb0a914d1d4: Deleting consensus metadata
I20250903 08:07:58.763084 15472 ts_tablet_manager.cc:1918] T 85020976e2fe4523bec20360a7b56599 P 4fe0343417e4407cacb543879bd8a7e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20250903 08:07:58.763072 14859 catalog_manager.cc:4928] TS 18892f6d43244679a593d7aad5877a09 (127.12.201.194:45871): tablet 85020976e2fe4523bec20360a7b56599 (table test-table [id=8f3846e7cff04b8ea18185526271d068]) successfully deleted
I20250903 08:07:58.763149 15472 log.cc:1199] T 85020976e2fe4523bec20360a7b56599 P 4fe0343417e4407cacb543879bd8a7e9: Deleting WAL directory at /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/minicluster-data/ts-0/wal/wals/85020976e2fe4523bec20360a7b56599
I20250903 08:07:58.763234 14859 catalog_manager.cc:4928] TS c35d28b86cb846d399e9beb0a914d1d4 (127.12.201.195:41255): tablet 85020976e2fe4523bec20360a7b56599 (table test-table [id=8f3846e7cff04b8ea18185526271d068]) successfully deleted
I20250903 08:07:58.763423 15472 ts_tablet_manager.cc:1939] T 85020976e2fe4523bec20360a7b56599 P 4fe0343417e4407cacb543879bd8a7e9: Deleting consensus metadata
I20250903 08:07:58.763942 14859 catalog_manager.cc:4928] TS 4fe0343417e4407cacb543879bd8a7e9 (127.12.201.193:39419): tablet 85020976e2fe4523bec20360a7b56599 (table test-table [id=8f3846e7cff04b8ea18185526271d068]) successfully deleted
Sep 03 08:07:58 dist-test-slave-27r9 krb5kdc[14827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886878, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 03 08:07:58 dist-test-slave-27r9 krb5kdc[14827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886878, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.201.254@KRBTEST.COM
I20250903 08:07:58.787827 15481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:58.776370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55342 (local address 127.12.201.254:36815)
0903 08:07:58.776610 (+ 240us) server_negotiation.cc:207] Beginning negotiation
0903 08:07:58.776614 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:07:58.776630 (+ 16us) server_negotiation.cc:408] Connection header received
0903 08:07:58.776735 (+ 105us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:07:58.776738 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:07:58.776784 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:07:58.776868 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:07:58.777596 (+ 728us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:58.778118 (+ 522us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:07:58.778686 (+ 568us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:58.778810 (+ 124us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:58.785318 (+ 6508us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:07:58.785337 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:07:58.785340 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:07:58.785365 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:07:58.786639 (+ 1274us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:07:58.786965 (+ 326us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:07:58.786967 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:07:58.786969 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:07:58.787044 (+ 75us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:07:58.787210 (+ 166us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:07:58.787212 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:07:58.787213 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:07:58.787361 (+ 148us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:07:58.787459 (+ 98us) server_negotiation.cc:1036] Waiting for connection context
0903 08:07:58.787599 (+ 140us) server_negotiation.cc:300] Negotiation successful
0903 08:07:58.787699 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":83,"threads_started":1}
I20250903 08:07:58.790225 14873 catalog_manager.cc:2232] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55342:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 08:07:58.790624 14873 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 08:07:58.794934 15014 tablet_service.cc:1468] Processing CreateTablet for tablet 03392b37dfd64b11b84bf0dae6d6a136 (DEFAULT_TABLE table=test-table [id=6c3cfa8eea6a4b0ba71a11ca679a60b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:07:58.794996 15156 tablet_service.cc:1468] Processing CreateTablet for tablet 03392b37dfd64b11b84bf0dae6d6a136 (DEFAULT_TABLE table=test-table [id=6c3cfa8eea6a4b0ba71a11ca679a60b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:07:58.795076 15298 tablet_service.cc:1468] Processing CreateTablet for tablet 03392b37dfd64b11b84bf0dae6d6a136 (DEFAULT_TABLE table=test-table [id=6c3cfa8eea6a4b0ba71a11ca679a60b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:07:58.795253 15156 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03392b37dfd64b11b84bf0dae6d6a136. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:07:58.795253 15014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03392b37dfd64b11b84bf0dae6d6a136. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:07:58.795364 15298 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03392b37dfd64b11b84bf0dae6d6a136. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:07:58.797138 15482 tablet_bootstrap.cc:492] T 03392b37dfd64b11b84bf0dae6d6a136 P c35d28b86cb846d399e9beb0a914d1d4: Bootstrap starting.
I20250903 08:07:58.797717 15484 tablet_bootstrap.cc:492] T 03392b37dfd64b11b84bf0dae6d6a136 P 4fe0343417e4407cacb543879bd8a7e9: Bootstrap starting.
I20250903 08:07:58.797956 15483 tablet_bootstrap.cc:492] T 03392b37dfd64b11b84bf0dae6d6a136 P 18892f6d43244679a593d7aad5877a09: Bootstrap starting.
I20250903 08:07:58.798499 15482 tablet_bootstrap.cc:654] T 03392b37dfd64b11b84bf0dae6d6a136 P c35d28b86cb846d399e9beb0a914d1d4: Neither blocks nor log segments found. Creating new log.
I20250903 08:07:58.798570 15484 tablet_bootstrap.cc:654] T 03392b37dfd64b11b84bf0dae6d6a136 P 4fe0343417e4407cacb543879bd8a7e9: Neither blocks nor log segments found. Creating new log.
I20250903 08:07:58.798913 15483 tablet_bootstrap.cc:654] T 03392b37dfd64b11b84bf0dae6d6a136 P 18892f6d43244679a593d7aad5877a09: Neither blocks nor log segments found. Creating new log.
I20250903 08:07:58.799753 15482 tablet_bootstrap.cc:492] T 03392b37dfd64b11b84bf0dae6d6a136 P c35d28b86cb846d399e9beb0a914d1d4: No bootstrap required, opened a new log
I20250903 08:07:58.799784 15484 tablet_bootstrap.cc:492] T 03392b37dfd64b11b84bf0dae6d6a136 P 4fe0343417e4407cacb543879bd8a7e9: No bootstrap required, opened a new log
I20250903 08:07:58.799841 15482 ts_tablet_manager.cc:1397] T 03392b37dfd64b11b84bf0dae6d6a136 P c35d28b86cb846d399e9beb0a914d1d4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250903 08:07:58.800012 15484 ts_tablet_manager.cc:1397] T 03392b37dfd64b11b84bf0dae6d6a136 P 4fe0343417e4407cacb543879bd8a7e9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250903 08:07:58.800087 15483 tablet_bootstrap.cc:492] T 03392b37dfd64b11b84bf0dae6d6a136 P 18892f6d43244679a593d7aad5877a09: No bootstrap required, opened a new log
I20250903 08:07:58.800163 15483 ts_tablet_manager.cc:1397] T 03392b37dfd64b11b84bf0dae6d6a136 P 18892f6d43244679a593d7aad5877a09: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250903 08:07:58.800288 15482 raft_consensus.cc:357] T 03392b37dfd64b11b84bf0dae6d6a136 P c35d28b86cb846d399e9beb0a914d1d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe0343417e4407cacb543879bd8a7e9" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 39419 } } peers { permanent_uuid: "18892f6d43244679a593d7aad5877a09" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 45871 } } peers { permanent_uuid: "c35d28b86cb846d399e9beb0a914d1d4" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 41255 } }
I20250903 08:07:58.800390 15482 raft_consensus.cc:383] T 03392b37dfd64b11b84bf0dae6d6a136 P c35d28b86cb846d399e9beb0a914d1d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:07:58.800423 15482 raft_consensus.cc:738] T 03392b37dfd64b11b84bf0dae6d6a136 P c35d28b86cb846d399e9beb0a914d1d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c35d28b86cb846d399e9beb0a914d1d4, State: Initialized, Role: FOLLOWER
I20250903 08:07:58.800463 15484 raft_consensus.cc:357] T 03392b37dfd64b11b84bf0dae6d6a136 P 4fe0343417e4407cacb543879bd8a7e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe0343417e4407cacb543879bd8a7e9" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 39419 } } peers { permanent_uuid: "18892f6d43244679a593d7aad5877a09" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 45871 } } peers { permanent_uuid: "c35d28b86cb846d399e9beb0a914d1d4" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 41255 } }
I20250903 08:07:58.800500 15483 raft_consensus.cc:357] T 03392b37dfd64b11b84bf0dae6d6a136 P 18892f6d43244679a593d7aad5877a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe0343417e4407cacb543879bd8a7e9" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 39419 } } peers { permanent_uuid: "18892f6d43244679a593d7aad5877a09" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 45871 } } peers { permanent_uuid: "c35d28b86cb846d399e9beb0a914d1d4" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 41255 } }
I20250903 08:07:58.800560 15484 raft_consensus.cc:383] T 03392b37dfd64b11b84bf0dae6d6a136 P 4fe0343417e4407cacb543879bd8a7e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:07:58.800577 15483 raft_consensus.cc:383] T 03392b37dfd64b11b84bf0dae6d6a136 P 18892f6d43244679a593d7aad5877a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:07:58.800547 15482 consensus_queue.cc:260] T 03392b37dfd64b11b84bf0dae6d6a136 P c35d28b86cb846d399e9beb0a914d1d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe0343417e4407cacb543879bd8a7e9" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 39419 } } peers { permanent_uuid: "18892f6d43244679a593d7aad5877a09" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 45871 } } peers { permanent_uuid: "c35d28b86cb846d399e9beb0a914d1d4" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 41255 } }
I20250903 08:07:58.800596 15484 raft_consensus.cc:738] T 03392b37dfd64b11b84bf0dae6d6a136 P 4fe0343417e4407cacb543879bd8a7e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fe0343417e4407cacb543879bd8a7e9, State: Initialized, Role: FOLLOWER
I20250903 08:07:58.800612 15483 raft_consensus.cc:738] T 03392b37dfd64b11b84bf0dae6d6a136 P 18892f6d43244679a593d7aad5877a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18892f6d43244679a593d7aad5877a09, State: Initialized, Role: FOLLOWER
I20250903 08:07:58.800727 15484 consensus_queue.cc:260] T 03392b37dfd64b11b84bf0dae6d6a136 P 4fe0343417e4407cacb543879bd8a7e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe0343417e4407cacb543879bd8a7e9" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 39419 } } peers { permanent_uuid: "18892f6d43244679a593d7aad5877a09" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 45871 } } peers { permanent_uuid: "c35d28b86cb846d399e9beb0a914d1d4" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 41255 } }
I20250903 08:07:58.800752 15483 consensus_queue.cc:260] T 03392b37dfd64b11b84bf0dae6d6a136 P 18892f6d43244679a593d7aad5877a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe0343417e4407cacb543879bd8a7e9" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 39419 } } peers { permanent_uuid: "18892f6d43244679a593d7aad5877a09" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 45871 } } peers { permanent_uuid: "c35d28b86cb846d399e9beb0a914d1d4" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 41255 } }
I20250903 08:07:58.800843 15482 ts_tablet_manager.cc:1428] T 03392b37dfd64b11b84bf0dae6d6a136 P c35d28b86cb846d399e9beb0a914d1d4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250903 08:07:58.801026 15484 ts_tablet_manager.cc:1428] T 03392b37dfd64b11b84bf0dae6d6a136 P 4fe0343417e4407cacb543879bd8a7e9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250903 08:07:58.801514 15483 ts_tablet_manager.cc:1428] T 03392b37dfd64b11b84bf0dae6d6a136 P 18892f6d43244679a593d7aad5877a09: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250903 08:07:58.927356 15488 raft_consensus.cc:491] T 03392b37dfd64b11b84bf0dae6d6a136 P c35d28b86cb846d399e9beb0a914d1d4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 08:07:58.927493 15488 raft_consensus.cc:513] T 03392b37dfd64b11b84bf0dae6d6a136 P c35d28b86cb846d399e9beb0a914d1d4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe0343417e4407cacb543879bd8a7e9" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 39419 } } peers { permanent_uuid: "18892f6d43244679a593d7aad5877a09" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 45871 } } peers { permanent_uuid: "c35d28b86cb846d399e9beb0a914d1d4" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 41255 } }
I20250903 08:07:58.927932 15488 leader_election.cc:290] T 03392b37dfd64b11b84bf0dae6d6a136 P c35d28b86cb846d399e9beb0a914d1d4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4fe0343417e4407cacb543879bd8a7e9 (127.12.201.193:39419), 18892f6d43244679a593d7aad5877a09 (127.12.201.194:45871)
I20250903 08:07:58.928414 15034 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "03392b37dfd64b11b84bf0dae6d6a136" candidate_uuid: "c35d28b86cb846d399e9beb0a914d1d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fe0343417e4407cacb543879bd8a7e9" is_pre_election: true
I20250903 08:07:58.928442 15176 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "03392b37dfd64b11b84bf0dae6d6a136" candidate_uuid: "c35d28b86cb846d399e9beb0a914d1d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18892f6d43244679a593d7aad5877a09" is_pre_election: true
I20250903 08:07:58.928546 15034 raft_consensus.cc:2466] T 03392b37dfd64b11b84bf0dae6d6a136 P 4fe0343417e4407cacb543879bd8a7e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c35d28b86cb846d399e9beb0a914d1d4 in term 0.
I20250903 08:07:58.928578 15176 raft_consensus.cc:2466] T 03392b37dfd64b11b84bf0dae6d6a136 P 18892f6d43244679a593d7aad5877a09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c35d28b86cb846d399e9beb0a914d1d4 in term 0.
I20250903 08:07:58.928875 15249 leader_election.cc:304] T 03392b37dfd64b11b84bf0dae6d6a136 P c35d28b86cb846d399e9beb0a914d1d4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fe0343417e4407cacb543879bd8a7e9, c35d28b86cb846d399e9beb0a914d1d4; no voters:
I20250903 08:07:58.929072 15488 raft_consensus.cc:2802] T 03392b37dfd64b11b84bf0dae6d6a136 P c35d28b86cb846d399e9beb0a914d1d4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 08:07:58.929128 15488 raft_consensus.cc:491] T 03392b37dfd64b11b84bf0dae6d6a136 P c35d28b86cb846d399e9beb0a914d1d4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 08:07:58.929183 15488 raft_consensus.cc:3058] T 03392b37dfd64b11b84bf0dae6d6a136 P c35d28b86cb846d399e9beb0a914d1d4 [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:07:58.930083 15488 raft_consensus.cc:513] T 03392b37dfd64b11b84bf0dae6d6a136 P c35d28b86cb846d399e9beb0a914d1d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe0343417e4407cacb543879bd8a7e9" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 39419 } } peers { permanent_uuid: "18892f6d43244679a593d7aad5877a09" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 45871 } } peers { permanent_uuid: "c35d28b86cb846d399e9beb0a914d1d4" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 41255 } }
I20250903 08:07:58.930387 15488 leader_election.cc:290] T 03392b37dfd64b11b84bf0dae6d6a136 P c35d28b86cb846d399e9beb0a914d1d4 [CANDIDATE]: Term 1 election: Requested vote from peers 4fe0343417e4407cacb543879bd8a7e9 (127.12.201.193:39419), 18892f6d43244679a593d7aad5877a09 (127.12.201.194:45871)
I20250903 08:07:58.930732 15034 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "03392b37dfd64b11b84bf0dae6d6a136" candidate_uuid: "c35d28b86cb846d399e9beb0a914d1d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fe0343417e4407cacb543879bd8a7e9"
I20250903 08:07:58.930752 15176 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "03392b37dfd64b11b84bf0dae6d6a136" candidate_uuid: "c35d28b86cb846d399e9beb0a914d1d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18892f6d43244679a593d7aad5877a09"
I20250903 08:07:58.930832 15034 raft_consensus.cc:3058] T 03392b37dfd64b11b84bf0dae6d6a136 P 4fe0343417e4407cacb543879bd8a7e9 [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:07:58.930853 15176 raft_consensus.cc:3058] T 03392b37dfd64b11b84bf0dae6d6a136 P 18892f6d43244679a593d7aad5877a09 [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:07:58.931671 15034 raft_consensus.cc:2466] T 03392b37dfd64b11b84bf0dae6d6a136 P 4fe0343417e4407cacb543879bd8a7e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c35d28b86cb846d399e9beb0a914d1d4 in term 1.
I20250903 08:07:58.931691 15176 raft_consensus.cc:2466] T 03392b37dfd64b11b84bf0dae6d6a136 P 18892f6d43244679a593d7aad5877a09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c35d28b86cb846d399e9beb0a914d1d4 in term 1.
I20250903 08:07:58.931950 15249 leader_election.cc:304] T 03392b37dfd64b11b84bf0dae6d6a136 P c35d28b86cb846d399e9beb0a914d1d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18892f6d43244679a593d7aad5877a09, c35d28b86cb846d399e9beb0a914d1d4; no voters:
I20250903 08:07:58.932106 15488 raft_consensus.cc:2802] T 03392b37dfd64b11b84bf0dae6d6a136 P c35d28b86cb846d399e9beb0a914d1d4 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 08:07:58.932305 15488 raft_consensus.cc:695] T 03392b37dfd64b11b84bf0dae6d6a136 P c35d28b86cb846d399e9beb0a914d1d4 [term 1 LEADER]: Becoming Leader. State: Replica: c35d28b86cb846d399e9beb0a914d1d4, State: Running, Role: LEADER
I20250903 08:07:58.932448 15488 consensus_queue.cc:237] T 03392b37dfd64b11b84bf0dae6d6a136 P c35d28b86cb846d399e9beb0a914d1d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe0343417e4407cacb543879bd8a7e9" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 39419 } } peers { permanent_uuid: "18892f6d43244679a593d7aad5877a09" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 45871 } } peers { permanent_uuid: "c35d28b86cb846d399e9beb0a914d1d4" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 41255 } }
I20250903 08:07:58.934016 14873 catalog_manager.cc:5582] T 03392b37dfd64b11b84bf0dae6d6a136 P c35d28b86cb846d399e9beb0a914d1d4 reported cstate change: term changed from 0 to 1, leader changed from <none> to c35d28b86cb846d399e9beb0a914d1d4 (127.12.201.195). New cstate: current_term: 1 leader_uuid: "c35d28b86cb846d399e9beb0a914d1d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe0343417e4407cacb543879bd8a7e9" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 39419 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18892f6d43244679a593d7aad5877a09" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 45871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c35d28b86cb846d399e9beb0a914d1d4" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 41255 } health_report { overall_health: HEALTHY } } }
I20250903 08:07:58.955468 15493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:58.953121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57010 (local address 127.12.201.195:41255)
0903 08:07:58.953363 (+ 242us) server_negotiation.cc:207] Beginning negotiation
0903 08:07:58.953366 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 08:07:58.953377 (+ 11us) server_negotiation.cc:408] Connection header received
0903 08:07:58.953413 (+ 36us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:07:58.953416 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:07:58.953456 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:07:58.953528 (+ 72us) server_negotiation.cc:227] Negotiated authn=TOKEN
0903 08:07:58.953795 (+ 267us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:58.954286 (+ 491us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:07:58.954840 (+ 554us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:58.954964 (+ 124us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:58.955055 (+ 91us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0903 08:07:58.955123 (+ 68us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0903 08:07:58.955182 (+ 59us) server_negotiation.cc:1036] Waiting for connection context
0903 08:07:58.955254 (+ 72us) server_negotiation.cc:300] Negotiation successful
0903 08:07:58.955296 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":96,"threads_started":1}
I20250903 08:07:58.960913 15034 raft_consensus.cc:1273] T 03392b37dfd64b11b84bf0dae6d6a136 P 4fe0343417e4407cacb543879bd8a7e9 [term 1 FOLLOWER]: Refusing update from remote peer c35d28b86cb846d399e9beb0a914d1d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 08:07:58.960953 15176 raft_consensus.cc:1273] T 03392b37dfd64b11b84bf0dae6d6a136 P 18892f6d43244679a593d7aad5877a09 [term 1 FOLLOWER]: Refusing update from remote peer c35d28b86cb846d399e9beb0a914d1d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 08:07:58.961352 15490 consensus_queue.cc:1035] T 03392b37dfd64b11b84bf0dae6d6a136 P c35d28b86cb846d399e9beb0a914d1d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fe0343417e4407cacb543879bd8a7e9" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 39419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 08:07:58.961483 15488 consensus_queue.cc:1035] T 03392b37dfd64b11b84bf0dae6d6a136 P c35d28b86cb846d399e9beb0a914d1d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18892f6d43244679a593d7aad5877a09" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 45871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 08:07:58.987870 14873 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55342:
table { table_name: "test-table" } modify_external_catalogs: true
I20250903 08:07:58.988039 14873 catalog_manager.cc:2730] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55342:
table { table_name: "test-table" } modify_external_catalogs: true
I20250903 08:07:58.989776 14873 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 911c1d9d6e52403ca64f3910291fda10: Sending DeleteTablet for 3 replicas of tablet 03392b37dfd64b11b84bf0dae6d6a136
I20250903 08:07:58.990325 15156 tablet_service.cc:1515] Processing DeleteTablet for tablet 03392b37dfd64b11b84bf0dae6d6a136 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 08:07:58 UTC) from {username='kudu', principal='kudu/127.12.201.254@KRBTEST.COM'} at 127.0.0.1:51956
I20250903 08:07:58.990355 15014 tablet_service.cc:1515] Processing DeleteTablet for tablet 03392b37dfd64b11b84bf0dae6d6a136 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 08:07:58 UTC) from {username='kudu', principal='kudu/127.12.201.254@KRBTEST.COM'} at 127.0.0.1:51148
I20250903 08:07:58.990355 15298 tablet_service.cc:1515] Processing DeleteTablet for tablet 03392b37dfd64b11b84bf0dae6d6a136 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 08:07:58 UTC) from {username='kudu', principal='kudu/127.12.201.254@KRBTEST.COM'} at 127.0.0.1:60350
I20250903 08:07:58.990527 15471 tablet_replica.cc:331] T 03392b37dfd64b11b84bf0dae6d6a136 P 18892f6d43244679a593d7aad5877a09: stopping tablet replica
I20250903 08:07:58.990566 15473 tablet_replica.cc:331] T 03392b37dfd64b11b84bf0dae6d6a136 P c35d28b86cb846d399e9beb0a914d1d4: stopping tablet replica
I20250903 08:07:58.990650 15471 raft_consensus.cc:2241] T 03392b37dfd64b11b84bf0dae6d6a136 P 18892f6d43244679a593d7aad5877a09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 08:07:58.990689 15473 raft_consensus.cc:2241] T 03392b37dfd64b11b84bf0dae6d6a136 P c35d28b86cb846d399e9beb0a914d1d4 [term 1 LEADER]: Raft consensus shutting down.
I20250903 08:07:58.990758 15471 raft_consensus.cc:2270] T 03392b37dfd64b11b84bf0dae6d6a136 P 18892f6d43244679a593d7aad5877a09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 08:07:58.990837 15473 raft_consensus.cc:2270] T 03392b37dfd64b11b84bf0dae6d6a136 P c35d28b86cb846d399e9beb0a914d1d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 08:07:58.990921 15472 tablet_replica.cc:331] T 03392b37dfd64b11b84bf0dae6d6a136 P 4fe0343417e4407cacb543879bd8a7e9: stopping tablet replica
I20250903 08:07:58.990952 14873 catalog_manager.cc:2232] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55342:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 08:07:58.991062 15472 raft_consensus.cc:2241] T 03392b37dfd64b11b84bf0dae6d6a136 P 4fe0343417e4407cacb543879bd8a7e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 08:07:58.991144 15472 raft_consensus.cc:2270] T 03392b37dfd64b11b84bf0dae6d6a136 P 4fe0343417e4407cacb543879bd8a7e9 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250903 08:07:58.991384 14873 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 08:07:58.991624 15473 ts_tablet_manager.cc:1905] T 03392b37dfd64b11b84bf0dae6d6a136 P c35d28b86cb846d399e9beb0a914d1d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 08:07:58.991899 15472 ts_tablet_manager.cc:1905] T 03392b37dfd64b11b84bf0dae6d6a136 P 4fe0343417e4407cacb543879bd8a7e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 08:07:58.993685 15473 ts_tablet_manager.cc:1918] T 03392b37dfd64b11b84bf0dae6d6a136 P c35d28b86cb846d399e9beb0a914d1d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20250903 08:07:58.993763 15473 log.cc:1199] T 03392b37dfd64b11b84bf0dae6d6a136 P c35d28b86cb846d399e9beb0a914d1d4: Deleting WAL directory at /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/minicluster-data/ts-2/wal/wals/03392b37dfd64b11b84bf0dae6d6a136
I20250903 08:07:58.994033 15473 ts_tablet_manager.cc:1939] T 03392b37dfd64b11b84bf0dae6d6a136 P c35d28b86cb846d399e9beb0a914d1d4: Deleting consensus metadata
I20250903 08:07:58.994046 15472 ts_tablet_manager.cc:1918] T 03392b37dfd64b11b84bf0dae6d6a136 P 4fe0343417e4407cacb543879bd8a7e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20250903 08:07:58.994097 15472 log.cc:1199] T 03392b37dfd64b11b84bf0dae6d6a136 P 4fe0343417e4407cacb543879bd8a7e9: Deleting WAL directory at /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/minicluster-data/ts-0/wal/wals/03392b37dfd64b11b84bf0dae6d6a136
I20250903 08:07:58.994313 15472 ts_tablet_manager.cc:1939] T 03392b37dfd64b11b84bf0dae6d6a136 P 4fe0343417e4407cacb543879bd8a7e9: Deleting consensus metadata
I20250903 08:07:58.994812 14859 catalog_manager.cc:4928] TS c35d28b86cb846d399e9beb0a914d1d4 (127.12.201.195:41255): tablet 03392b37dfd64b11b84bf0dae6d6a136 (table test-table [id=6c3cfa8eea6a4b0ba71a11ca679a60b2]) successfully deleted
I20250903 08:07:58.994969 14859 catalog_manager.cc:4928] TS 4fe0343417e4407cacb543879bd8a7e9 (127.12.201.193:39419): tablet 03392b37dfd64b11b84bf0dae6d6a136 (table test-table [id=6c3cfa8eea6a4b0ba71a11ca679a60b2]) successfully deleted
I20250903 08:07:58.995266 15471 ts_tablet_manager.cc:1905] T 03392b37dfd64b11b84bf0dae6d6a136 P 18892f6d43244679a593d7aad5877a09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 08:07:58.995841 15014 tablet_service.cc:1468] Processing CreateTablet for tablet 2ca4ad16056148a0b1bd6a8a9dd7762d (DEFAULT_TABLE table=test-table [id=ec55e6d67e55455ea873b00146fed11f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:07:58.996042 15014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ca4ad16056148a0b1bd6a8a9dd7762d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:07:58.997123 15471 ts_tablet_manager.cc:1918] T 03392b37dfd64b11b84bf0dae6d6a136 P 18892f6d43244679a593d7aad5877a09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20250903 08:07:58.997191 15471 log.cc:1199] T 03392b37dfd64b11b84bf0dae6d6a136 P 18892f6d43244679a593d7aad5877a09: Deleting WAL directory at /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/minicluster-data/ts-1/wal/wals/03392b37dfd64b11b84bf0dae6d6a136
I20250903 08:07:58.997416 15471 ts_tablet_manager.cc:1939] T 03392b37dfd64b11b84bf0dae6d6a136 P 18892f6d43244679a593d7aad5877a09: Deleting consensus metadata
I20250903 08:07:58.997500 15484 tablet_bootstrap.cc:492] T 2ca4ad16056148a0b1bd6a8a9dd7762d P 4fe0343417e4407cacb543879bd8a7e9: Bootstrap starting.
I20250903 08:07:58.997956 15156 tablet_service.cc:1468] Processing CreateTablet for tablet 2ca4ad16056148a0b1bd6a8a9dd7762d (DEFAULT_TABLE table=test-table [id=ec55e6d67e55455ea873b00146fed11f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:07:58.998179 15156 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ca4ad16056148a0b1bd6a8a9dd7762d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:07:58.998157 15298 tablet_service.cc:1468] Processing CreateTablet for tablet 2ca4ad16056148a0b1bd6a8a9dd7762d (DEFAULT_TABLE table=test-table [id=ec55e6d67e55455ea873b00146fed11f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:07:58.998389 15298 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ca4ad16056148a0b1bd6a8a9dd7762d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:07:58.998660 15484 tablet_bootstrap.cc:654] T 2ca4ad16056148a0b1bd6a8a9dd7762d P 4fe0343417e4407cacb543879bd8a7e9: Neither blocks nor log segments found. Creating new log.
I20250903 08:07:58.999845 15484 tablet_bootstrap.cc:492] T 2ca4ad16056148a0b1bd6a8a9dd7762d P 4fe0343417e4407cacb543879bd8a7e9: No bootstrap required, opened a new log
I20250903 08:07:58.999926 15484 ts_tablet_manager.cc:1397] T 2ca4ad16056148a0b1bd6a8a9dd7762d P 4fe0343417e4407cacb543879bd8a7e9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250903 08:07:59.000344 15482 tablet_bootstrap.cc:492] T 2ca4ad16056148a0b1bd6a8a9dd7762d P c35d28b86cb846d399e9beb0a914d1d4: Bootstrap starting.
I20250903 08:07:59.000419 15484 raft_consensus.cc:357] T 2ca4ad16056148a0b1bd6a8a9dd7762d P 4fe0343417e4407cacb543879bd8a7e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe0343417e4407cacb543879bd8a7e9" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 39419 } } peers { permanent_uuid: "18892f6d43244679a593d7aad5877a09" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 45871 } } peers { permanent_uuid: "c35d28b86cb846d399e9beb0a914d1d4" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 41255 } }
I20250903 08:07:59.000532 15484 raft_consensus.cc:383] T 2ca4ad16056148a0b1bd6a8a9dd7762d P 4fe0343417e4407cacb543879bd8a7e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:07:59.000569 15484 raft_consensus.cc:738] T 2ca4ad16056148a0b1bd6a8a9dd7762d P 4fe0343417e4407cacb543879bd8a7e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fe0343417e4407cacb543879bd8a7e9, State: Initialized, Role: FOLLOWER
I20250903 08:07:59.000653 14859 catalog_manager.cc:4928] TS 18892f6d43244679a593d7aad5877a09 (127.12.201.194:45871): tablet 03392b37dfd64b11b84bf0dae6d6a136 (table test-table [id=6c3cfa8eea6a4b0ba71a11ca679a60b2]) successfully deleted
I20250903 08:07:59.000690 15484 consensus_queue.cc:260] T 2ca4ad16056148a0b1bd6a8a9dd7762d P 4fe0343417e4407cacb543879bd8a7e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe0343417e4407cacb543879bd8a7e9" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 39419 } } peers { permanent_uuid: "18892f6d43244679a593d7aad5877a09" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 45871 } } peers { permanent_uuid: "c35d28b86cb846d399e9beb0a914d1d4" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 41255 } }
I20250903 08:07:59.000942 15484 ts_tablet_manager.cc:1428] T 2ca4ad16056148a0b1bd6a8a9dd7762d P 4fe0343417e4407cacb543879bd8a7e9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250903 08:07:59.001109 15482 tablet_bootstrap.cc:654] T 2ca4ad16056148a0b1bd6a8a9dd7762d P c35d28b86cb846d399e9beb0a914d1d4: Neither blocks nor log segments found. Creating new log.
I20250903 08:07:59.001138 15483 tablet_bootstrap.cc:492] T 2ca4ad16056148a0b1bd6a8a9dd7762d P 18892f6d43244679a593d7aad5877a09: Bootstrap starting.
I20250903 08:07:59.001842 15483 tablet_bootstrap.cc:654] T 2ca4ad16056148a0b1bd6a8a9dd7762d P 18892f6d43244679a593d7aad5877a09: Neither blocks nor log segments found. Creating new log.
I20250903 08:07:59.002528 15482 tablet_bootstrap.cc:492] T 2ca4ad16056148a0b1bd6a8a9dd7762d P c35d28b86cb846d399e9beb0a914d1d4: No bootstrap required, opened a new log
I20250903 08:07:59.002605 15482 ts_tablet_manager.cc:1397] T 2ca4ad16056148a0b1bd6a8a9dd7762d P c35d28b86cb846d399e9beb0a914d1d4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250903 08:07:59.002821 15483 tablet_bootstrap.cc:492] T 2ca4ad16056148a0b1bd6a8a9dd7762d P 18892f6d43244679a593d7aad5877a09: No bootstrap required, opened a new log
I20250903 08:07:59.002897 15483 ts_tablet_manager.cc:1397] T 2ca4ad16056148a0b1bd6a8a9dd7762d P 18892f6d43244679a593d7aad5877a09: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250903 08:07:59.003003 15482 raft_consensus.cc:357] T 2ca4ad16056148a0b1bd6a8a9dd7762d P c35d28b86cb846d399e9beb0a914d1d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe0343417e4407cacb543879bd8a7e9" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 39419 } } peers { permanent_uuid: "18892f6d43244679a593d7aad5877a09" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 45871 } } peers { permanent_uuid: "c35d28b86cb846d399e9beb0a914d1d4" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 41255 } }
I20250903 08:07:59.003106 15482 raft_consensus.cc:383] T 2ca4ad16056148a0b1bd6a8a9dd7762d P c35d28b86cb846d399e9beb0a914d1d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:07:59.003131 15482 raft_consensus.cc:738] T 2ca4ad16056148a0b1bd6a8a9dd7762d P c35d28b86cb846d399e9beb0a914d1d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c35d28b86cb846d399e9beb0a914d1d4, State: Initialized, Role: FOLLOWER
I20250903 08:07:59.003234 15482 consensus_queue.cc:260] T 2ca4ad16056148a0b1bd6a8a9dd7762d P c35d28b86cb846d399e9beb0a914d1d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe0343417e4407cacb543879bd8a7e9" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 39419 } } peers { permanent_uuid: "18892f6d43244679a593d7aad5877a09" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 45871 } } peers { permanent_uuid: "c35d28b86cb846d399e9beb0a914d1d4" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 41255 } }
I20250903 08:07:59.003322 15483 raft_consensus.cc:357] T 2ca4ad16056148a0b1bd6a8a9dd7762d P 18892f6d43244679a593d7aad5877a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe0343417e4407cacb543879bd8a7e9" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 39419 } } peers { permanent_uuid: "18892f6d43244679a593d7aad5877a09" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 45871 } } peers { permanent_uuid: "c35d28b86cb846d399e9beb0a914d1d4" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 41255 } }
I20250903 08:07:59.003417 15483 raft_consensus.cc:383] T 2ca4ad16056148a0b1bd6a8a9dd7762d P 18892f6d43244679a593d7aad5877a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:07:59.003458 15483 raft_consensus.cc:738] T 2ca4ad16056148a0b1bd6a8a9dd7762d P 18892f6d43244679a593d7aad5877a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18892f6d43244679a593d7aad5877a09, State: Initialized, Role: FOLLOWER
I20250903 08:07:59.003473 15482 ts_tablet_manager.cc:1428] T 2ca4ad16056148a0b1bd6a8a9dd7762d P c35d28b86cb846d399e9beb0a914d1d4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250903 08:07:59.003571 15483 consensus_queue.cc:260] T 2ca4ad16056148a0b1bd6a8a9dd7762d P 18892f6d43244679a593d7aad5877a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe0343417e4407cacb543879bd8a7e9" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 39419 } } peers { permanent_uuid: "18892f6d43244679a593d7aad5877a09" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 45871 } } peers { permanent_uuid: "c35d28b86cb846d399e9beb0a914d1d4" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 41255 } }
I20250903 08:07:59.003841 15483 ts_tablet_manager.cc:1428] T 2ca4ad16056148a0b1bd6a8a9dd7762d P 18892f6d43244679a593d7aad5877a09: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250903 08:07:59.009001 15489 raft_consensus.cc:491] T 2ca4ad16056148a0b1bd6a8a9dd7762d P 4fe0343417e4407cacb543879bd8a7e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 08:07:59.009090 15489 raft_consensus.cc:513] T 2ca4ad16056148a0b1bd6a8a9dd7762d P 4fe0343417e4407cacb543879bd8a7e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe0343417e4407cacb543879bd8a7e9" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 39419 } } peers { permanent_uuid: "18892f6d43244679a593d7aad5877a09" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 45871 } } peers { permanent_uuid: "c35d28b86cb846d399e9beb0a914d1d4" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 41255 } }
I20250903 08:07:59.010006 15489 leader_election.cc:290] T 2ca4ad16056148a0b1bd6a8a9dd7762d P 4fe0343417e4407cacb543879bd8a7e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18892f6d43244679a593d7aad5877a09 (127.12.201.194:45871), c35d28b86cb846d399e9beb0a914d1d4 (127.12.201.195:41255)
I20250903 08:07:59.013691 15503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:59.010529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.195:41255 (local address 127.12.201.193:36659)
0903 08:07:59.010866 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0903 08:07:59.010878 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0903 08:07:59.010962 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:07:59.011264 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:07:59.011267 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:07:59.011305 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 08:07:59.011558 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:59.011566 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:59.012423 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:07:59.012426 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:07:59.013257 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:59.013263 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:59.013379 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:59.013401 (+ 22us) client_negotiation.cc:769] Sending connection context
0903 08:07:59.013453 (+ 52us) client_negotiation.cc:240] Negotiation successful
0903 08:07:59.013511 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":89,"threads_started":1}
I20250903 08:07:59.013921 15501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:59.010217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.194:45871 (local address 127.12.201.193:56693)
0903 08:07:59.010458 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0903 08:07:59.010474 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0903 08:07:59.010575 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:07:59.010952 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:07:59.010955 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:07:59.010969 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 08:07:59.011275 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:59.011285 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:59.012311 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:07:59.012314 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:07:59.013254 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:59.013263 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:59.013380 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:59.013401 (+ 21us) client_negotiation.cc:769] Sending connection context
0903 08:07:59.013453 (+ 52us) client_negotiation.cc:240] Negotiation successful
0903 08:07:59.013511 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":76,"threads_started":1}
I20250903 08:07:59.014113 15502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:59.010263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.193:56693 (local address 127.12.201.194:45871)
0903 08:07:59.010502 (+ 239us) server_negotiation.cc:207] Beginning negotiation
0903 08:07:59.010507 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 08:07:59.010526 (+ 19us) server_negotiation.cc:408] Connection header received
0903 08:07:59.010683 (+ 157us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:07:59.010687 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:07:59.010737 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:07:59.010827 (+ 90us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 08:07:59.011397 (+ 570us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:59.012217 (+ 820us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:07:59.013388 (+ 1171us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:59.013843 (+ 455us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:59.013879 (+ 36us) server_negotiation.cc:1036] Waiting for connection context
0903 08:07:59.013930 (+ 51us) server_negotiation.cc:300] Negotiation successful
0903 08:07:59.013983 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":77,"threads_started":1}
I20250903 08:07:59.014137 15493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:59.010604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.193:36659 (local address 127.12.201.195:41255)
0903 08:07:59.010721 (+ 117us) server_negotiation.cc:207] Beginning negotiation
0903 08:07:59.010725 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:07:59.010930 (+ 205us) server_negotiation.cc:408] Connection header received
0903 08:07:59.011116 (+ 186us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:07:59.011119 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:07:59.011168 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:07:59.011254 (+ 86us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 08:07:59.011666 (+ 412us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:59.012338 (+ 672us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:07:59.013378 (+ 1040us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:59.013923 (+ 545us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:59.013953 (+ 30us) server_negotiation.cc:1036] Waiting for connection context
0903 08:07:59.013993 (+ 40us) server_negotiation.cc:300] Negotiation successful
0903 08:07:59.014043 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20250903 08:07:59.014504 15318 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2ca4ad16056148a0b1bd6a8a9dd7762d" candidate_uuid: "4fe0343417e4407cacb543879bd8a7e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c35d28b86cb846d399e9beb0a914d1d4" is_pre_election: true
I20250903 08:07:59.014631 15318 raft_consensus.cc:2466] T 2ca4ad16056148a0b1bd6a8a9dd7762d P c35d28b86cb846d399e9beb0a914d1d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fe0343417e4407cacb543879bd8a7e9 in term 0.
I20250903 08:07:59.014658 15176 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2ca4ad16056148a0b1bd6a8a9dd7762d" candidate_uuid: "4fe0343417e4407cacb543879bd8a7e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18892f6d43244679a593d7aad5877a09" is_pre_election: true
I20250903 08:07:59.014766 15176 raft_consensus.cc:2466] T 2ca4ad16056148a0b1bd6a8a9dd7762d P 18892f6d43244679a593d7aad5877a09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fe0343417e4407cacb543879bd8a7e9 in term 0.
I20250903 08:07:59.015051 14965 leader_election.cc:304] T 2ca4ad16056148a0b1bd6a8a9dd7762d P 4fe0343417e4407cacb543879bd8a7e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fe0343417e4407cacb543879bd8a7e9, c35d28b86cb846d399e9beb0a914d1d4; no voters:
I20250903 08:07:59.015278 15489 raft_consensus.cc:2802] T 2ca4ad16056148a0b1bd6a8a9dd7762d P 4fe0343417e4407cacb543879bd8a7e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 08:07:59.015328 15489 raft_consensus.cc:491] T 2ca4ad16056148a0b1bd6a8a9dd7762d P 4fe0343417e4407cacb543879bd8a7e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 08:07:59.015369 15489 raft_consensus.cc:3058] T 2ca4ad16056148a0b1bd6a8a9dd7762d P 4fe0343417e4407cacb543879bd8a7e9 [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:07:59.016162 15489 raft_consensus.cc:513] T 2ca4ad16056148a0b1bd6a8a9dd7762d P 4fe0343417e4407cacb543879bd8a7e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe0343417e4407cacb543879bd8a7e9" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 39419 } } peers { permanent_uuid: "18892f6d43244679a593d7aad5877a09" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 45871 } } peers { permanent_uuid: "c35d28b86cb846d399e9beb0a914d1d4" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 41255 } }
I20250903 08:07:59.016433 15489 leader_election.cc:290] T 2ca4ad16056148a0b1bd6a8a9dd7762d P 4fe0343417e4407cacb543879bd8a7e9 [CANDIDATE]: Term 1 election: Requested vote from peers 18892f6d43244679a593d7aad5877a09 (127.12.201.194:45871), c35d28b86cb846d399e9beb0a914d1d4 (127.12.201.195:41255)
I20250903 08:07:59.016716 15176 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2ca4ad16056148a0b1bd6a8a9dd7762d" candidate_uuid: "4fe0343417e4407cacb543879bd8a7e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18892f6d43244679a593d7aad5877a09"
I20250903 08:07:59.016808 15318 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2ca4ad16056148a0b1bd6a8a9dd7762d" candidate_uuid: "4fe0343417e4407cacb543879bd8a7e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c35d28b86cb846d399e9beb0a914d1d4"
I20250903 08:07:59.016820 15176 raft_consensus.cc:3058] T 2ca4ad16056148a0b1bd6a8a9dd7762d P 18892f6d43244679a593d7aad5877a09 [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:07:59.016893 15318 raft_consensus.cc:3058] T 2ca4ad16056148a0b1bd6a8a9dd7762d P c35d28b86cb846d399e9beb0a914d1d4 [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:07:59.017608 15176 raft_consensus.cc:2466] T 2ca4ad16056148a0b1bd6a8a9dd7762d P 18892f6d43244679a593d7aad5877a09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fe0343417e4407cacb543879bd8a7e9 in term 1.
I20250903 08:07:59.017632 15318 raft_consensus.cc:2466] T 2ca4ad16056148a0b1bd6a8a9dd7762d P c35d28b86cb846d399e9beb0a914d1d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fe0343417e4407cacb543879bd8a7e9 in term 1.
I20250903 08:07:59.017868 14965 leader_election.cc:304] T 2ca4ad16056148a0b1bd6a8a9dd7762d P 4fe0343417e4407cacb543879bd8a7e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fe0343417e4407cacb543879bd8a7e9, c35d28b86cb846d399e9beb0a914d1d4; no voters:
I20250903 08:07:59.018008 15489 raft_consensus.cc:2802] T 2ca4ad16056148a0b1bd6a8a9dd7762d P 4fe0343417e4407cacb543879bd8a7e9 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 08:07:59.018185 15489 raft_consensus.cc:695] T 2ca4ad16056148a0b1bd6a8a9dd7762d P 4fe0343417e4407cacb543879bd8a7e9 [term 1 LEADER]: Becoming Leader. State: Replica: 4fe0343417e4407cacb543879bd8a7e9, State: Running, Role: LEADER
I20250903 08:07:59.018417 15489 consensus_queue.cc:237] T 2ca4ad16056148a0b1bd6a8a9dd7762d P 4fe0343417e4407cacb543879bd8a7e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe0343417e4407cacb543879bd8a7e9" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 39419 } } peers { permanent_uuid: "18892f6d43244679a593d7aad5877a09" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 45871 } } peers { permanent_uuid: "c35d28b86cb846d399e9beb0a914d1d4" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 41255 } }
I20250903 08:07:59.020296 14873 catalog_manager.cc:5582] T 2ca4ad16056148a0b1bd6a8a9dd7762d P 4fe0343417e4407cacb543879bd8a7e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4fe0343417e4407cacb543879bd8a7e9 (127.12.201.193). New cstate: current_term: 1 leader_uuid: "4fe0343417e4407cacb543879bd8a7e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe0343417e4407cacb543879bd8a7e9" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 39419 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "18892f6d43244679a593d7aad5877a09" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 45871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c35d28b86cb846d399e9beb0a914d1d4" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 41255 } health_report { overall_health: UNKNOWN } } }
I20250903 08:07:59.035727 15506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:59.032715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48546 (local address 127.12.201.193:39419)
0903 08:07:59.032968 (+ 253us) server_negotiation.cc:207] Beginning negotiation
0903 08:07:59.032971 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 08:07:59.032980 (+ 9us) server_negotiation.cc:408] Connection header received
0903 08:07:59.033048 (+ 68us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:07:59.033050 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:07:59.033096 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:07:59.033192 (+ 96us) server_negotiation.cc:227] Negotiated authn=TOKEN
0903 08:07:59.033515 (+ 323us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:59.034020 (+ 505us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:07:59.034608 (+ 588us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:59.034797 (+ 189us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:59.034849 (+ 52us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0903 08:07:59.035345 (+ 496us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0903 08:07:59.035448 (+ 103us) server_negotiation.cc:1036] Waiting for connection context
0903 08:07:59.035526 (+ 78us) server_negotiation.cc:300] Negotiation successful
0903 08:07:59.035573 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":87,"threads_started":1}
I20250903 08:07:59.043861 15509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:59.041467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.194:45871 (local address 127.12.201.193:48115)
0903 08:07:59.041736 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0903 08:07:59.041748 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0903 08:07:59.041833 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:07:59.042032 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:07:59.042034 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:07:59.042040 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0903 08:07:59.042233 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:59.042237 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:59.042813 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:07:59.042815 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:07:59.043294 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:59.043302 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:59.043386 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:59.043417 (+ 31us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0903 08:07:59.043660 (+ 243us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0903 08:07:59.043664 (+ 4us) client_negotiation.cc:769] Sending connection context
0903 08:07:59.043707 (+ 43us) client_negotiation.cc:240] Negotiation successful
0903 08:07:59.043751 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":81,"threads_started":1}
I20250903 08:07:59.043856 15502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:59.041641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.193:48115 (local address 127.12.201.194:45871)
0903 08:07:59.041756 (+ 115us) server_negotiation.cc:207] Beginning negotiation
0903 08:07:59.041759 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 08:07:59.041790 (+ 31us) server_negotiation.cc:408] Connection header received
0903 08:07:59.041918 (+ 128us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:07:59.041920 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:07:59.041958 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:07:59.042017 (+ 59us) server_negotiation.cc:227] Negotiated authn=TOKEN
0903 08:07:59.042318 (+ 301us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:59.042733 (+ 415us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:07:59.043383 (+ 650us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:59.043501 (+ 118us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:59.043536 (+ 35us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0903 08:07:59.043595 (+ 59us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0903 08:07:59.043651 (+ 56us) server_negotiation.cc:1036] Waiting for connection context
0903 08:07:59.043730 (+ 79us) server_negotiation.cc:300] Negotiation successful
0903 08:07:59.043766 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20250903 08:07:59.049188 15318 raft_consensus.cc:1273] T 2ca4ad16056148a0b1bd6a8a9dd7762d P c35d28b86cb846d399e9beb0a914d1d4 [term 1 FOLLOWER]: Refusing update from remote peer 4fe0343417e4407cacb543879bd8a7e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 08:07:59.049178 15176 raft_consensus.cc:1273] T 2ca4ad16056148a0b1bd6a8a9dd7762d P 18892f6d43244679a593d7aad5877a09 [term 1 FOLLOWER]: Refusing update from remote peer 4fe0343417e4407cacb543879bd8a7e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 08:07:59.049590 15504 consensus_queue.cc:1035] T 2ca4ad16056148a0b1bd6a8a9dd7762d P 4fe0343417e4407cacb543879bd8a7e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18892f6d43244679a593d7aad5877a09" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 45871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 08:07:59.049741 15489 consensus_queue.cc:1035] T 2ca4ad16056148a0b1bd6a8a9dd7762d P 4fe0343417e4407cacb543879bd8a7e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c35d28b86cb846d399e9beb0a914d1d4" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 41255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 08:07:59.064854 15506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:59.061768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.194:58135 (local address 127.12.201.193:39419)
0903 08:07:59.061900 (+ 132us) server_negotiation.cc:207] Beginning negotiation
0903 08:07:59.061905 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 08:07:59.062113 (+ 208us) server_negotiation.cc:408] Connection header received
0903 08:07:59.062270 (+ 157us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:07:59.062273 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:07:59.062318 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:07:59.062379 (+ 61us) server_negotiation.cc:227] Negotiated authn=TOKEN
0903 08:07:59.062674 (+ 295us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:59.063562 (+ 888us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:07:59.064226 (+ 664us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:07:59.064404 (+ 178us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:59.064455 (+ 51us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0903 08:07:59.064521 (+ 66us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0903 08:07:59.064592 (+ 71us) server_negotiation.cc:1036] Waiting for connection context
0903 08:07:59.064712 (+ 120us) server_negotiation.cc:300] Negotiation successful
0903 08:07:59.064752 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20250903 08:07:59.064873 15514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:07:59.061695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.193:39419 (local address 127.12.201.194:58135)
0903 08:07:59.062039 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0903 08:07:59.062055 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0903 08:07:59.062181 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:07:59.062399 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:07:59.062401 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:07:59.062407 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0903 08:07:59.062597 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:59.062602 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:59.063663 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:07:59.063667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:07:59.064117 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:07:59.064123 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:07:59.064207 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:07:59.064230 (+ 23us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0903 08:07:59.064610 (+ 380us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0903 08:07:59.064618 (+ 8us) client_negotiation.cc:769] Sending connection context
0903 08:07:59.064693 (+ 75us) client_negotiation.cc:240] Negotiation successful
0903 08:07:59.064763 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":89,"threads_started":1}
W20250903 08:07:59.065469 15456 tablet_replica.cc:1304] Aborted: operation has been aborted: cancelling pending write operations
I20250903 08:08:00.617597 14873 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55342:
table { table_name: "test-table" } modify_external_catalogs: true
I20250903 08:08:00.617743 14873 catalog_manager.cc:2730] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55342:
table { table_name: "test-table" } modify_external_catalogs: true
I20250903 08:08:00.620034 14873 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 911c1d9d6e52403ca64f3910291fda10: Sending DeleteTablet for 3 replicas of tablet 2ca4ad16056148a0b1bd6a8a9dd7762d
I20250903 08:08:00.620564 15014 tablet_service.cc:1515] Processing DeleteTablet for tablet 2ca4ad16056148a0b1bd6a8a9dd7762d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 08:08:00 UTC) from {username='kudu', principal='kudu/127.12.201.254@KRBTEST.COM'} at 127.0.0.1:51148
I20250903 08:08:00.620636 15156 tablet_service.cc:1515] Processing DeleteTablet for tablet 2ca4ad16056148a0b1bd6a8a9dd7762d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 08:08:00 UTC) from {username='kudu', principal='kudu/127.12.201.254@KRBTEST.COM'} at 127.0.0.1:51956
I20250903 08:08:00.620671 15298 tablet_service.cc:1515] Processing DeleteTablet for tablet 2ca4ad16056148a0b1bd6a8a9dd7762d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 08:08:00 UTC) from {username='kudu', principal='kudu/127.12.201.254@KRBTEST.COM'} at 127.0.0.1:60350
I20250903 08:08:00.620981 15539 tablet_replica.cc:331] T 2ca4ad16056148a0b1bd6a8a9dd7762d P c35d28b86cb846d399e9beb0a914d1d4: stopping tablet replica
I20250903 08:08:00.621026 13095 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUML90r/build/debug/bin/kudu with pid 14949
I20250903 08:08:00.621747 15538 tablet_replica.cc:331] T 2ca4ad16056148a0b1bd6a8a9dd7762d P 18892f6d43244679a593d7aad5877a09: stopping tablet replica
I20250903 08:08:00.621776 15539 raft_consensus.cc:2241] T 2ca4ad16056148a0b1bd6a8a9dd7762d P c35d28b86cb846d399e9beb0a914d1d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 08:08:00.621865 15538 raft_consensus.cc:2241] T 2ca4ad16056148a0b1bd6a8a9dd7762d P 18892f6d43244679a593d7aad5877a09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 08:08:00.621898 15539 raft_consensus.cc:2270] T 2ca4ad16056148a0b1bd6a8a9dd7762d P c35d28b86cb846d399e9beb0a914d1d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 08:08:00.621984 15538 raft_consensus.cc:2270] T 2ca4ad16056148a0b1bd6a8a9dd7762d P 18892f6d43244679a593d7aad5877a09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 08:08:00.622581 15539 ts_tablet_manager.cc:1905] T 2ca4ad16056148a0b1bd6a8a9dd7762d P c35d28b86cb846d399e9beb0a914d1d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 08:08:00.622612 15538 ts_tablet_manager.cc:1905] T 2ca4ad16056148a0b1bd6a8a9dd7762d P 18892f6d43244679a593d7aad5877a09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 08:08:00.625128 15539 ts_tablet_manager.cc:1918] T 2ca4ad16056148a0b1bd6a8a9dd7762d P c35d28b86cb846d399e9beb0a914d1d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20250903 08:08:00.625128 15538 ts_tablet_manager.cc:1918] T 2ca4ad16056148a0b1bd6a8a9dd7762d P 18892f6d43244679a593d7aad5877a09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20250903 08:08:00.625211 15539 log.cc:1199] T 2ca4ad16056148a0b1bd6a8a9dd7762d P c35d28b86cb846d399e9beb0a914d1d4: Deleting WAL directory at /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/minicluster-data/ts-2/wal/wals/2ca4ad16056148a0b1bd6a8a9dd7762d
I20250903 08:08:00.625221 15538 log.cc:1199] T 2ca4ad16056148a0b1bd6a8a9dd7762d P 18892f6d43244679a593d7aad5877a09: Deleting WAL directory at /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756886860253814-13095-0/minicluster-data/ts-1/wal/wals/2ca4ad16056148a0b1bd6a8a9dd7762d
I20250903 08:08:00.625509 15538 ts_tablet_manager.cc:1939] T 2ca4ad16056148a0b1bd6a8a9dd7762d P 18892f6d43244679a593d7aad5877a09: Deleting consensus metadata
I20250903 08:08:00.625509 15539 ts_tablet_manager.cc:1939] T 2ca4ad16056148a0b1bd6a8a9dd7762d P c35d28b86cb846d399e9beb0a914d1d4: Deleting consensus metadata
I20250903 08:08:00.626242 14859 catalog_manager.cc:4928] TS c35d28b86cb846d399e9beb0a914d1d4 (127.12.201.195:41255): tablet 2ca4ad16056148a0b1bd6a8a9dd7762d (table test-table [id=ec55e6d67e55455ea873b00146fed11f]) successfully deleted
I20250903 08:08:00.626425 14859 catalog_manager.cc:4928] TS 18892f6d43244679a593d7aad5877a09 (127.12.201.194:45871): tablet 2ca4ad16056148a0b1bd6a8a9dd7762d (table test-table [id=ec55e6d67e55455ea873b00146fed11f]) successfully deleted
W20250903 08:08:00.629689 14859 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.201.193:39419 (error 108)
I20250903 08:08:00.629993 13095 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUML90r/build/debug/bin/kudu with pid 15091
I20250903 08:08:00.631749 15541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:00.630940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.193:39419 (local address 127.0.0.1:48554)
0903 08:08:00.631604 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:00.631657 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.201.193:39419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":99,"threads_started":1}
W20250903 08:08:00.631886 14859 catalog_manager.cc:4671] TS 4fe0343417e4407cacb543879bd8a7e9 (127.12.201.193:39419): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2ca4ad16056148a0b1bd6a8a9dd7762d: Network error: Client connection negotiation failed: client connection to 127.12.201.193:39419: connect: Connection refused (error 111)
I20250903 08:08:00.637290 13095 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUML90r/build/debug/bin/kudu with pid 15233
I20250903 08:08:00.643379 13095 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUML90r/build/debug/bin/kudu with pid 14843
2025-09-03T08:08:00Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (9817 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 03 08:08:00 dist-test-slave-27r9 krb5kdc[15544](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 03 08:08:00 dist-test-slave-27r9 krb5kdc[15544](info): set up 2 sockets
Sep 03 08:08:00 dist-test-slave-27r9 krb5kdc[15544](info): commencing operation
krb5kdc: starting...
W20250903 08:08:02.619359 13095 mini_kdc.cc:121] Time spent starting KDC: real 1.953s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/krb5kdc/test-user.keytab.
Sep 03 08:08:02 dist-test-slave-27r9 krb5kdc[15544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886882, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-09-03T08:08:02Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-03T08:08:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.201.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.201.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.201.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.201.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.201.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.201.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.201.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.201.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/krb5kdc/kudu.keytab.
I20250903 08:08:02.747671 13095 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUML90r/build/debug/bin/kudu
/tmp/dist-test-taskUML90r/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.201.254:37207
--webserver_interface=127.12.201.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.201.212:45957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.201.254:37207
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 08:08:02.849225 15560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:08:02.849434 15560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:08:02.849512 15560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:08:02.852543 15560 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250903 08:08:02.852597 15560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 08:08:02.852633 15560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:08:02.852667 15560 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250903 08:08:02.852706 15560 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250903 08:08:02.856667 15560 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.201.212:45957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.201.254:37207
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.201.254:37207
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.201.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15560
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 07:50:08 UTC on 5fd53c4cbb9d
build id 7874
I20250903 08:08:02.857616 15560 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:08:02.858412 15560 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:08:02.863409 15565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:02.863431 15566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:08:02.863536 15560 server_base.cc:1047] running on GCE node
W20250903 08:08:02.863433 15568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:08:02.864089 15560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:08:02.864881 15560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:08:02.866036 15560 hybrid_clock.cc:648] HybridClock initialized: now 1756886882866030 us; error 26 us; skew 500 ppm
Sep 03 08:08:02 dist-test-slave-27r9 krb5kdc[15544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886882, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 08:08:02.868526 15560 init.cc:377] Logged in from keytab as kudu/127.12.201.254@KRBTEST.COM (short username kudu)
I20250903 08:08:02.869489 15560 webserver.cc:480] Webserver started at http://127.12.201.254:45325/ using document root <none> and password file <none>
I20250903 08:08:02.869990 15560 fs_manager.cc:362] Metadata directory not provided
I20250903 08:08:02.870043 15560 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:08:02.870213 15560 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:08:02.871850 15560 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/minicluster-data/master-0/data/instance:
uuid: "4518c1e2d9e24c2aae69f4d4632e70fe"
format_stamp: "Formatted at 2025-09-03 08:08:02 on dist-test-slave-27r9"
server_key: "f7cfd0734dd928af11afbfa6583c491e"
server_key_iv: "f34c4cbd775bc13ef8eb3fd64a66e1e4"
server_key_version: "encryptionkey@0"
I20250903 08:08:02.872300 15560 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/minicluster-data/master-0/wal/instance:
uuid: "4518c1e2d9e24c2aae69f4d4632e70fe"
format_stamp: "Formatted at 2025-09-03 08:08:02 on dist-test-slave-27r9"
server_key: "f7cfd0734dd928af11afbfa6583c491e"
server_key_iv: "f34c4cbd775bc13ef8eb3fd64a66e1e4"
server_key_version: "encryptionkey@0"
I20250903 08:08:02.875491 15560 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250903 08:08:02.877460 15575 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:02.878407 15560 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250903 08:08:02.878506 15560 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/minicluster-data/master-0/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/minicluster-data/master-0/wal
uuid: "4518c1e2d9e24c2aae69f4d4632e70fe"
format_stamp: "Formatted at 2025-09-03 08:08:02 on dist-test-slave-27r9"
server_key: "f7cfd0734dd928af11afbfa6583c491e"
server_key_iv: "f34c4cbd775bc13ef8eb3fd64a66e1e4"
server_key_version: "encryptionkey@0"
I20250903 08:08:02.878589 15560 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:08:02.888255 15560 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:08:02.890570 15560 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:08:02.890727 15560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:08:02.897339 15560 rpc_server.cc:307] RPC server started. Bound to: 127.12.201.254:37207
I20250903 08:08:02.897346 15627 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.201.254:37207 every 8 connection(s)
I20250903 08:08:02.898249 15560 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/minicluster-data/master-0/data/info.pb
I20250903 08:08:02.900502 15628 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:08:02.903108 13095 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUML90r/build/debug/bin/kudu as pid 15560
I20250903 08:08:02.903239 13095 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/minicluster-data/master-0/wal/instance
I20250903 08:08:02.903479 13095 external_mini_cluster.cc:1468] Setting key dde5fa5967f302853b85958c72166334
I20250903 08:08:02.906154 15628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4518c1e2d9e24c2aae69f4d4632e70fe: Bootstrap starting.
I20250903 08:08:02.908464 15628 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4518c1e2d9e24c2aae69f4d4632e70fe: Neither blocks nor log segments found. Creating new log.
I20250903 08:08:02.909231 15628 log.cc:826] T 00000000000000000000000000000000 P 4518c1e2d9e24c2aae69f4d4632e70fe: Log is configured to *not* fsync() on all Append() calls
Sep 03 08:08:02 dist-test-slave-27r9 krb5kdc[15544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886882, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.201.254@KRBTEST.COM
I20250903 08:08:02.910889 15628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4518c1e2d9e24c2aae69f4d4632e70fe: No bootstrap required, opened a new log
I20250903 08:08:02.913033 15628 raft_consensus.cc:357] T 00000000000000000000000000000000 P 4518c1e2d9e24c2aae69f4d4632e70fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4518c1e2d9e24c2aae69f4d4632e70fe" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 37207 } }
I20250903 08:08:02.913211 15628 raft_consensus.cc:383] T 00000000000000000000000000000000 P 4518c1e2d9e24c2aae69f4d4632e70fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:08:02.913291 15628 raft_consensus.cc:738] T 00000000000000000000000000000000 P 4518c1e2d9e24c2aae69f4d4632e70fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4518c1e2d9e24c2aae69f4d4632e70fe, State: Initialized, Role: FOLLOWER
I20250903 08:08:02.913705 15628 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4518c1e2d9e24c2aae69f4d4632e70fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4518c1e2d9e24c2aae69f4d4632e70fe" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 37207 } }
I20250903 08:08:02.913828 15628 raft_consensus.cc:397] T 00000000000000000000000000000000 P 4518c1e2d9e24c2aae69f4d4632e70fe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250903 08:08:02.913887 15628 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4518c1e2d9e24c2aae69f4d4632e70fe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250903 08:08:02.913995 15628 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 4518c1e2d9e24c2aae69f4d4632e70fe [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:08:02.914776 15628 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4518c1e2d9e24c2aae69f4d4632e70fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4518c1e2d9e24c2aae69f4d4632e70fe" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 37207 } }
I20250903 08:08:02.915104 15628 leader_election.cc:304] T 00000000000000000000000000000000 P 4518c1e2d9e24c2aae69f4d4632e70fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4518c1e2d9e24c2aae69f4d4632e70fe; no voters:
I20250903 08:08:02.915246 15631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:02.904610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35014 (local address 127.12.201.254:37207)
0903 08:08:02.904961 (+ 351us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:02.904969 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:02.905001 (+ 32us) server_negotiation.cc:408] Connection header received
0903 08:08:02.905487 (+ 486us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:02.905502 (+ 15us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:02.905759 (+ 257us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:02.906002 (+ 243us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:02.907060 (+ 1058us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:02.907799 (+ 739us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:02.908643 (+ 844us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:02.908827 (+ 184us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:02.911103 (+ 2276us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:02.911123 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:02.911133 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:08:02.911162 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:02.912800 (+ 1638us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:02.913206 (+ 406us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:02.913210 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:02.913214 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:02.913326 (+ 112us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:02.913587 (+ 261us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:02.913590 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:02.913591 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:02.913885 (+ 294us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:02.914088 (+ 203us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:02.914327 (+ 239us) server_negotiation.cc:300] Negotiation successful
0903 08:08:02.914557 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":114,"threads_started":1}
I20250903 08:08:02.915349 15628 leader_election.cc:290] T 00000000000000000000000000000000 P 4518c1e2d9e24c2aae69f4d4632e70fe [CANDIDATE]: Term 1 election: Requested vote from peers
I20250903 08:08:02.915894 15633 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 4518c1e2d9e24c2aae69f4d4632e70fe [term 1 FOLLOWER]: Leader election won for term 1
I20250903 08:08:02.916101 15633 raft_consensus.cc:695] T 00000000000000000000000000000000 P 4518c1e2d9e24c2aae69f4d4632e70fe [term 1 LEADER]: Becoming Leader. State: Replica: 4518c1e2d9e24c2aae69f4d4632e70fe, State: Running, Role: LEADER
I20250903 08:08:02.916396 15633 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4518c1e2d9e24c2aae69f4d4632e70fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4518c1e2d9e24c2aae69f4d4632e70fe" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 37207 } }
I20250903 08:08:02.917593 15628 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4518c1e2d9e24c2aae69f4d4632e70fe [sys.catalog]: configured and running, proceeding with master startup.
I20250903 08:08:02.917982 15634 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4518c1e2d9e24c2aae69f4d4632e70fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4518c1e2d9e24c2aae69f4d4632e70fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4518c1e2d9e24c2aae69f4d4632e70fe" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 37207 } } }
I20250903 08:08:02.918125 15634 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4518c1e2d9e24c2aae69f4d4632e70fe [sys.catalog]: This master's current role is: LEADER
I20250903 08:08:02.918143 15635 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4518c1e2d9e24c2aae69f4d4632e70fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4518c1e2d9e24c2aae69f4d4632e70fe. Latest consensus state: current_term: 1 leader_uuid: "4518c1e2d9e24c2aae69f4d4632e70fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4518c1e2d9e24c2aae69f4d4632e70fe" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 37207 } } }
I20250903 08:08:02.918211 15635 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4518c1e2d9e24c2aae69f4d4632e70fe [sys.catalog]: This master's current role is: LEADER
I20250903 08:08:02.918927 15637 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250903 08:08:02.921808 15637 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250903 08:08:02.926416 15637 catalog_manager.cc:1349] Generated new cluster ID: c06d87df96894904977ec024a3ce0883
I20250903 08:08:02.926476 15637 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250903 08:08:02.938401 15637 catalog_manager.cc:1372] Generated new certificate authority record
I20250903 08:08:02.939517 15637 catalog_manager.cc:1506] Loading token signing keys...
I20250903 08:08:02.954932 15637 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 4518c1e2d9e24c2aae69f4d4632e70fe: Generated new TSK 0
I20250903 08:08:02.955637 15637 catalog_manager.cc:1516] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.201.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.201.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.201.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.201.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.201.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.201.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.201.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.201.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/krb5kdc/kudu.keytab.
I20250903 08:08:03.003568 13095 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUML90r/build/debug/bin/kudu
/tmp/dist-test-taskUML90r/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.201.193:0
--local_ip_for_outbound_sockets=127.12.201.193
--webserver_interface=127.12.201.193
--webserver_port=0
--tserver_master_addrs=127.12.201.254:37207
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.201.212:45957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 08:08:03.106302 15656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:08:03.106523 15656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:08:03.106604 15656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:08:03.109731 15656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 08:08:03.109790 15656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:08:03.109917 15656 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.201.193
I20250903 08:08:03.113867 15656 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.201.212:45957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.201.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.201.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.12.201.254:37207
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15656
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.201.193
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 07:50:08 UTC on 5fd53c4cbb9d
build id 7874
I20250903 08:08:03.114861 15656 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:08:03.115700 15656 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:08:03.121951 15662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:03.121991 15664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:03.121961 15661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:08:03.122128 15656 server_base.cc:1047] running on GCE node
I20250903 08:08:03.122496 15656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:08:03.123051 15656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:08:03.124197 15656 hybrid_clock.cc:648] HybridClock initialized: now 1756886883124186 us; error 29 us; skew 500 ppm
Sep 03 08:08:03 dist-test-slave-27r9 krb5kdc[15544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886883, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 08:08:03.126781 15656 init.cc:377] Logged in from keytab as kudu/127.12.201.193@KRBTEST.COM (short username kudu)
I20250903 08:08:03.127745 15656 webserver.cc:480] Webserver started at http://127.12.201.193:43557/ using document root <none> and password file <none>
I20250903 08:08:03.128261 15656 fs_manager.cc:362] Metadata directory not provided
I20250903 08:08:03.128307 15656 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:08:03.128453 15656 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:08:03.130059 15656 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-0/data/instance:
uuid: "4e39cf4b81f94b8898e374cb9df66ae5"
format_stamp: "Formatted at 2025-09-03 08:08:03 on dist-test-slave-27r9"
server_key: "b1da31c427d4d116d78d38395be70389"
server_key_iv: "de081bf29f2f263f8b26c660cbdda1ac"
server_key_version: "encryptionkey@0"
I20250903 08:08:03.130462 15656 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-0/wal/instance:
uuid: "4e39cf4b81f94b8898e374cb9df66ae5"
format_stamp: "Formatted at 2025-09-03 08:08:03 on dist-test-slave-27r9"
server_key: "b1da31c427d4d116d78d38395be70389"
server_key_iv: "de081bf29f2f263f8b26c660cbdda1ac"
server_key_version: "encryptionkey@0"
I20250903 08:08:03.133697 15656 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20250903 08:08:03.135725 15671 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:03.136657 15656 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20250903 08:08:03.136747 15656 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-0/wal
uuid: "4e39cf4b81f94b8898e374cb9df66ae5"
format_stamp: "Formatted at 2025-09-03 08:08:03 on dist-test-slave-27r9"
server_key: "b1da31c427d4d116d78d38395be70389"
server_key_iv: "de081bf29f2f263f8b26c660cbdda1ac"
server_key_version: "encryptionkey@0"
I20250903 08:08:03.136821 15656 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:08:03.158777 15656 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:08:03.161661 15656 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:08:03.161828 15656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:08:03.162309 15656 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 08:08:03.163134 15656 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 08:08:03.163182 15656 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:03.163220 15656 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 08:08:03.163240 15656 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:03.171761 15656 rpc_server.cc:307] RPC server started. Bound to: 127.12.201.193:40899
I20250903 08:08:03.171797 15784 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.201.193:40899 every 8 connection(s)
I20250903 08:08:03.172670 15656 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-0/data/info.pb
I20250903 08:08:03.179122 13095 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUML90r/build/debug/bin/kudu as pid 15656
I20250903 08:08:03.179265 13095 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-0/wal/instance
I20250903 08:08:03.179529 13095 external_mini_cluster.cc:1468] Setting key 9bf01bee0dfefb3cfda7121371cd29a3
Sep 03 08:08:03 dist-test-slave-27r9 krb5kdc[15544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886883, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.193@KRBTEST.COM for kudu/127.12.201.254@KRBTEST.COM
I20250903 08:08:03.185135 15631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:03.174316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.193:42411 (local address 127.12.201.254:37207)
0903 08:08:03.174460 (+ 144us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:03.174464 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:03.175100 (+ 636us) server_negotiation.cc:408] Connection header received
0903 08:08:03.175874 (+ 774us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:03.175877 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:03.175923 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:03.176001 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:03.177216 (+ 1215us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:03.177721 (+ 505us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:03.178766 (+ 1045us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:03.178971 (+ 205us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:03.181240 (+ 2269us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:03.181255 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:03.181257 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:08:03.181279 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:03.183145 (+ 1866us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:03.183724 (+ 579us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:03.183726 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:03.183728 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:03.183768 (+ 40us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:03.184073 (+ 305us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:03.184075 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:03.184077 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:03.184217 (+ 140us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:03.184304 (+ 87us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:03.184894 (+ 590us) server_negotiation.cc:300] Negotiation successful
0903 08:08:03.185014 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20250903 08:08:03.186094 15787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:03.174553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:37207 (local address 127.12.201.193:42411)
0903 08:08:03.174934 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:03.175006 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:03.175706 (+ 700us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:03.176144 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:03.176152 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:03.176528 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:08:03.177078 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:03.177089 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:03.177809 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:03.177812 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:03.178671 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:03.178678 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:03.178855 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:03.179552 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:08:03.179585 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:08:03.181095 (+ 1510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:08:03.183262 (+ 2167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:03.183271 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:03.183285 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:03.183613 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:03.183863 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:03.183869 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:03.183874 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:03.183988 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:03.184327 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:08:03.184335 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:08:03.184643 (+ 308us) client_negotiation.cc:769] Sending connection context
0903 08:08:03.184872 (+ 229us) client_negotiation.cc:240] Negotiation successful
0903 08:08:03.185158 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":96,"threads_started":1}
I20250903 08:08:03.187299 15785 heartbeater.cc:344] Connected to a master server at 127.12.201.254:37207
I20250903 08:08:03.187532 15785 heartbeater.cc:461] Registering TS with master...
I20250903 08:08:03.188055 15785 heartbeater.cc:507] Master 127.12.201.254:37207 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.12.201.194@KRBTEST.COM; defaulting to no policy
I20250903 08:08:03.189500 15592 ts_manager.cc:194] Registered new tserver with Master: 4e39cf4b81f94b8898e374cb9df66ae5 (127.12.201.193:40899)
I20250903 08:08:03.190701 15592 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.201.193@KRBTEST.COM'} at 127.12.201.193:42411
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.201.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.201.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.201.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.201.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.201.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.201.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.201.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/krb5kdc/kudu.keytab.
I20250903 08:08:03.223284 13095 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUML90r/build/debug/bin/kudu
/tmp/dist-test-taskUML90r/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.201.194:0
--local_ip_for_outbound_sockets=127.12.201.194
--webserver_interface=127.12.201.194
--webserver_port=0
--tserver_master_addrs=127.12.201.254:37207
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.201.212:45957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 08:08:03.326448 15792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:08:03.326659 15792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:08:03.326752 15792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:08:03.329919 15792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 08:08:03.329979 15792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:08:03.330086 15792 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.201.194
I20250903 08:08:03.334086 15792 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.201.212:45957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.201.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.201.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.12.201.254:37207
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15792
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.201.194
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 07:50:08 UTC on 5fd53c4cbb9d
build id 7874
I20250903 08:08:03.335098 15792 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:08:03.335929 15792 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:08:03.341979 15797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:03.341982 15798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:03.342279 15800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:08:03.342267 15792 server_base.cc:1047] running on GCE node
I20250903 08:08:03.342669 15792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:08:03.343197 15792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:08:03.344337 15792 hybrid_clock.cc:648] HybridClock initialized: now 1756886883344323 us; error 28 us; skew 500 ppm
Sep 03 08:08:03 dist-test-slave-27r9 krb5kdc[15544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886883, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 08:08:03.346843 15792 init.cc:377] Logged in from keytab as kudu/127.12.201.194@KRBTEST.COM (short username kudu)
I20250903 08:08:03.347846 15792 webserver.cc:480] Webserver started at http://127.12.201.194:34261/ using document root <none> and password file <none>
I20250903 08:08:03.348362 15792 fs_manager.cc:362] Metadata directory not provided
I20250903 08:08:03.348416 15792 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:08:03.348589 15792 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:08:03.350232 15792 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-1/data/instance:
uuid: "78581d5d676641f39951f001743e4d6e"
format_stamp: "Formatted at 2025-09-03 08:08:03 on dist-test-slave-27r9"
server_key: "b5d4aff2347f09a5662a48c2c9f87ba0"
server_key_iv: "06f7d449fd3af2d15a509989d5ddd7fb"
server_key_version: "encryptionkey@0"
I20250903 08:08:03.350680 15792 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-1/wal/instance:
uuid: "78581d5d676641f39951f001743e4d6e"
format_stamp: "Formatted at 2025-09-03 08:08:03 on dist-test-slave-27r9"
server_key: "b5d4aff2347f09a5662a48c2c9f87ba0"
server_key_iv: "06f7d449fd3af2d15a509989d5ddd7fb"
server_key_version: "encryptionkey@0"
I20250903 08:08:03.353901 15792 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20250903 08:08:03.355976 15807 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:03.356874 15792 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250903 08:08:03.356992 15792 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-1/wal
uuid: "78581d5d676641f39951f001743e4d6e"
format_stamp: "Formatted at 2025-09-03 08:08:03 on dist-test-slave-27r9"
server_key: "b5d4aff2347f09a5662a48c2c9f87ba0"
server_key_iv: "06f7d449fd3af2d15a509989d5ddd7fb"
server_key_version: "encryptionkey@0"
I20250903 08:08:03.357072 15792 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:08:03.386507 15792 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:08:03.389278 15792 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:08:03.389452 15792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:08:03.389971 15792 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 08:08:03.390766 15792 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 08:08:03.390815 15792 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:03.390872 15792 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 08:08:03.390906 15792 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:03.399689 15792 rpc_server.cc:307] RPC server started. Bound to: 127.12.201.194:41057
I20250903 08:08:03.399737 15920 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.201.194:41057 every 8 connection(s)
I20250903 08:08:03.400592 15792 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-1/data/info.pb
Sep 03 08:08:03 dist-test-slave-27r9 krb5kdc[15544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886883, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.194@KRBTEST.COM for kudu/127.12.201.254@KRBTEST.COM
I20250903 08:08:03.408547 13095 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUML90r/build/debug/bin/kudu as pid 15792
I20250903 08:08:03.408671 13095 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-1/wal/instance
I20250903 08:08:03.408914 13095 external_mini_cluster.cc:1468] Setting key 9ffe85d81e55238f4c0062e8e3d2518a
I20250903 08:08:03.412182 15631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:03.402181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.194:43447 (local address 127.12.201.254:37207)
0903 08:08:03.402349 (+ 168us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:03.402353 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:03.402966 (+ 613us) server_negotiation.cc:408] Connection header received
0903 08:08:03.403762 (+ 796us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:03.403765 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:03.403814 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:03.403888 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:03.405057 (+ 1169us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:03.405540 (+ 483us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:03.406546 (+ 1006us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:03.406669 (+ 123us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:03.408797 (+ 2128us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:03.408816 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:03.408818 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:08:03.408844 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:03.410394 (+ 1550us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:03.410837 (+ 443us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:03.410842 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:03.410844 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:03.410884 (+ 40us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:03.411194 (+ 310us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:03.411198 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:03.411200 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:03.411338 (+ 138us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:03.411429 (+ 91us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:03.411902 (+ 473us) server_negotiation.cc:300] Negotiation successful
0903 08:08:03.412021 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20250903 08:08:03.412770 15923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:03.402444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:37207 (local address 127.12.201.194:43447)
0903 08:08:03.402827 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:03.402861 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:03.403579 (+ 718us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:03.404017 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:03.404024 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:03.404398 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:08:03.404938 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:03.404951 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:03.405625 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:03.405628 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:03.406462 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:03.406469 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:03.406611 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:03.407179 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:08:03.407206 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:08:03.408635 (+ 1429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:08:03.410500 (+ 1865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:03.410505 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:03.410516 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:03.410743 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:03.410970 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:03.410973 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:03.411010 (+ 37us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:03.411095 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:03.411443 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:08:03.411451 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:08:03.411657 (+ 206us) client_negotiation.cc:769] Sending connection context
0903 08:08:03.411834 (+ 177us) client_negotiation.cc:240] Negotiation successful
0903 08:08:03.412094 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":96,"threads_started":1}
I20250903 08:08:03.413887 15921 heartbeater.cc:344] Connected to a master server at 127.12.201.254:37207
I20250903 08:08:03.414137 15921 heartbeater.cc:461] Registering TS with master...
I20250903 08:08:03.414825 15921 heartbeater.cc:507] Master 127.12.201.254:37207 requested a full tablet report, sending...
I20250903 08:08:03.415971 15592 ts_manager.cc:194] Registered new tserver with Master: 78581d5d676641f39951f001743e4d6e (127.12.201.194:41057)
I20250903 08:08:03.416736 15592 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.201.194@KRBTEST.COM'} at 127.12.201.194:43447
WARNING: no policy specified for kudu/127.12.201.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.201.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.201.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.201.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.201.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.201.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.201.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.201.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/krb5kdc/kudu.keytab.
I20250903 08:08:03.453671 13095 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUML90r/build/debug/bin/kudu
/tmp/dist-test-taskUML90r/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.201.195:0
--local_ip_for_outbound_sockets=127.12.201.195
--webserver_interface=127.12.201.195
--webserver_port=0
--tserver_master_addrs=127.12.201.254:37207
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.201.212:45957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 08:08:03.557894 15928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:08:03.558111 15928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:08:03.558176 15928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:08:03.561444 15928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 08:08:03.561506 15928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:08:03.561630 15928 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.201.195
I20250903 08:08:03.565639 15928 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.201.212:45957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.201.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.201.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.12.201.254:37207
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15928
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.201.195
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 07:50:08 UTC on 5fd53c4cbb9d
build id 7874
I20250903 08:08:03.566648 15928 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:08:03.567502 15928 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:08:03.573666 15933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:03.573666 15934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:08:03.573809 15928 server_base.cc:1047] running on GCE node
W20250903 08:08:03.573694 15936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:08:03.574312 15928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:08:03.574795 15928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:08:03.575938 15928 hybrid_clock.cc:648] HybridClock initialized: now 1756886883575930 us; error 31 us; skew 500 ppm
Sep 03 08:08:03 dist-test-slave-27r9 krb5kdc[15544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886883, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 08:08:03.578552 15928 init.cc:377] Logged in from keytab as kudu/127.12.201.195@KRBTEST.COM (short username kudu)
I20250903 08:08:03.579505 15928 webserver.cc:480] Webserver started at http://127.12.201.195:39803/ using document root <none> and password file <none>
I20250903 08:08:03.580034 15928 fs_manager.cc:362] Metadata directory not provided
I20250903 08:08:03.580082 15928 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:08:03.580237 15928 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:08:03.581825 15928 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-2/data/instance:
uuid: "e46f0236f0674797a3002c7e40080991"
format_stamp: "Formatted at 2025-09-03 08:08:03 on dist-test-slave-27r9"
server_key: "a748b3035b11be176cae1eae73a6ad30"
server_key_iv: "a23b17f260523596430bade88907f36f"
server_key_version: "encryptionkey@0"
I20250903 08:08:03.582288 15928 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-2/wal/instance:
uuid: "e46f0236f0674797a3002c7e40080991"
format_stamp: "Formatted at 2025-09-03 08:08:03 on dist-test-slave-27r9"
server_key: "a748b3035b11be176cae1eae73a6ad30"
server_key_iv: "a23b17f260523596430bade88907f36f"
server_key_version: "encryptionkey@0"
I20250903 08:08:03.585549 15928 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250903 08:08:03.587698 15943 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:03.593392 15928 fs_manager.cc:730] Time spent opening block manager: real 0.007s user 0.002s sys 0.000s
I20250903 08:08:03.593500 15928 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-2/wal
uuid: "e46f0236f0674797a3002c7e40080991"
format_stamp: "Formatted at 2025-09-03 08:08:03 on dist-test-slave-27r9"
server_key: "a748b3035b11be176cae1eae73a6ad30"
server_key_iv: "a23b17f260523596430bade88907f36f"
server_key_version: "encryptionkey@0"
I20250903 08:08:03.593621 15928 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:08:03.604292 15928 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:08:03.606721 15928 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:08:03.606864 15928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:08:03.607406 15928 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 08:08:03.608168 15928 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 08:08:03.608218 15928 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:03.608263 15928 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 08:08:03.608297 15928 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:03.616464 15928 rpc_server.cc:307] RPC server started. Bound to: 127.12.201.195:45801
I20250903 08:08:03.616497 16056 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.201.195:45801 every 8 connection(s)
I20250903 08:08:03.617328 15928 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-2/data/info.pb
I20250903 08:08:03.618682 13095 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUML90r/build/debug/bin/kudu as pid 15928
I20250903 08:08:03.618778 13095 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-2/wal/instance
I20250903 08:08:03.619056 13095 external_mini_cluster.cc:1468] Setting key 8d629929713b943d46843484598c871a
Sep 03 08:08:03 dist-test-slave-27r9 krb5kdc[15544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886883, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.195@KRBTEST.COM for kudu/127.12.201.254@KRBTEST.COM
I20250903 08:08:03.628726 15631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:03.620234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.195:51599 (local address 127.12.201.254:37207)
0903 08:08:03.620376 (+ 142us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:03.620381 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:03.620395 (+ 14us) server_negotiation.cc:408] Connection header received
0903 08:08:03.620749 (+ 354us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:03.620753 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:03.620806 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:03.620881 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:03.622067 (+ 1186us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:03.622563 (+ 496us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:03.623684 (+ 1121us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:03.623860 (+ 176us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:03.625949 (+ 2089us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:03.625966 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:03.625968 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:08:03.625992 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:03.627256 (+ 1264us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:03.627654 (+ 398us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:03.627657 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:03.627658 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:03.627694 (+ 36us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:03.627905 (+ 211us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:03.627907 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:03.627908 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:03.628041 (+ 133us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:03.628112 (+ 71us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:03.628518 (+ 406us) server_negotiation.cc:300] Negotiation successful
0903 08:08:03.628606 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20250903 08:08:03.629354 16059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:03.619302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:37207 (local address 127.12.201.195:51599)
0903 08:08:03.619733 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:03.619776 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:03.620571 (+ 795us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:03.621030 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:03.621040 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:03.621423 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:08:03.621931 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:03.621944 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:03.622673 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:03.622676 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:03.623572 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:03.623580 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:03.623763 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:03.624418 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:08:03.624442 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:08:03.625808 (+ 1366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:08:03.627350 (+ 1542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:03.627354 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:03.627363 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:03.627580 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:03.627756 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:03.627758 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:03.627760 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:03.627844 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:03.628122 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:08:03.628127 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:08:03.628334 (+ 207us) client_negotiation.cc:769] Sending connection context
0903 08:08:03.628495 (+ 161us) client_negotiation.cc:240] Negotiation successful
0903 08:08:03.628699 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":117,"threads_started":1}
I20250903 08:08:03.630301 16057 heartbeater.cc:344] Connected to a master server at 127.12.201.254:37207
I20250903 08:08:03.630518 16057 heartbeater.cc:461] Registering TS with master...
I20250903 08:08:03.630954 16057 heartbeater.cc:507] Master 127.12.201.254:37207 requested a full tablet report, sending...
I20250903 08:08:03.631897 15592 ts_manager.cc:194] Registered new tserver with Master: e46f0236f0674797a3002c7e40080991 (127.12.201.195:45801)
I20250903 08:08:03.632516 15592 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.201.195@KRBTEST.COM'} at 127.12.201.195:51599
I20250903 08:08:03.632601 13095 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20250903 08:08:03.643522 15631 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 08:08:03.642268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35030 (local address 127.12.201.254:37207)
0903 08:08:03.642403 (+ 135us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:03.642408 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:03.642530 (+ 122us) server_negotiation.cc:408] Connection header received
0903 08:08:03.642657 (+ 127us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:03.642663 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:03.642703 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:03.642770 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:03.643433 (+ 663us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35030: BlockingRecv error: recv got EOF from 127.0.0.1:35030 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20250903 08:08:03.644642 13095 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUML90r/build/debug/bin/kudu with pid 15656
I20250903 08:08:03.650005 13095 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUML90r/build/debug/bin/kudu with pid 15792
I20250903 08:08:03.655256 13095 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUML90r/build/debug/bin/kudu with pid 15928
I20250903 08:08:03.660362 13095 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUML90r/build/debug/bin/kudu with pid 15560
2025-09-03T08:08:03Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3015 ms)
[ RUN ] SecurityITest.TestRebalanceReportUnauthorized
Loading random data
Initializing database '/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 03 08:08:03 dist-test-slave-27r9 krb5kdc[16069](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 03 08:08:03 dist-test-slave-27r9 krb5kdc[16069](info): set up 2 sockets
Sep 03 08:08:03 dist-test-slave-27r9 krb5kdc[16069](info): commencing operation
krb5kdc: starting...
W20250903 08:08:05.635506 13095 mini_kdc.cc:121] Time spent starting KDC: real 1.954s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/krb5kdc/test-user.keytab.
Sep 03 08:08:05 dist-test-slave-27r9 krb5kdc[16069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886885, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-09-03T08:08:05Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-03T08:08:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.201.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.201.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.201.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.201.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.201.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.201.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.201.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.201.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/krb5kdc/kudu.keytab.
I20250903 08:08:05.765836 13095 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUML90r/build/debug/bin/kudu
/tmp/dist-test-taskUML90r/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.201.254:46375
--webserver_interface=127.12.201.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.201.212:39421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.201.254:46375
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 08:08:05.866577 16085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:08:05.866801 16085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:08:05.866855 16085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:08:05.870185 16085 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250903 08:08:05.870253 16085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 08:08:05.870277 16085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:08:05.870306 16085 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250903 08:08:05.870339 16085 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250903 08:08:05.874583 16085 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.201.212:39421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.201.254:46375
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.201.254:46375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.201.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16085
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 07:50:08 UTC on 5fd53c4cbb9d
build id 7874
I20250903 08:08:05.875622 16085 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:08:05.876447 16085 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:08:05.881785 16093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:05.881850 16091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:05.881906 16090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:08:05.882068 16085 server_base.cc:1047] running on GCE node
I20250903 08:08:05.882532 16085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:08:05.883409 16085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:08:05.884546 16085 hybrid_clock.cc:648] HybridClock initialized: now 1756886885884532 us; error 25 us; skew 500 ppm
Sep 03 08:08:05 dist-test-slave-27r9 krb5kdc[16069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886885, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 08:08:05.886826 16085 init.cc:377] Logged in from keytab as kudu/127.12.201.254@KRBTEST.COM (short username kudu)
I20250903 08:08:05.887820 16085 webserver.cc:480] Webserver started at http://127.12.201.254:34749/ using document root <none> and password file <none>
I20250903 08:08:05.888321 16085 fs_manager.cc:362] Metadata directory not provided
I20250903 08:08:05.888373 16085 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:08:05.888546 16085 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:08:05.890175 16085 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/minicluster-data/master-0/data/instance:
uuid: "1970879255e148dfa0e77a084962857b"
format_stamp: "Formatted at 2025-09-03 08:08:05 on dist-test-slave-27r9"
server_key: "5514c2e176313fe6013060c9773f53a6"
server_key_iv: "f7da0afae38361edef328a24739e1a6b"
server_key_version: "encryptionkey@0"
I20250903 08:08:05.890628 16085 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/minicluster-data/master-0/wal/instance:
uuid: "1970879255e148dfa0e77a084962857b"
format_stamp: "Formatted at 2025-09-03 08:08:05 on dist-test-slave-27r9"
server_key: "5514c2e176313fe6013060c9773f53a6"
server_key_iv: "f7da0afae38361edef328a24739e1a6b"
server_key_version: "encryptionkey@0"
I20250903 08:08:05.893882 16085 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250903 08:08:05.895972 16100 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:05.896895 16085 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250903 08:08:05.897009 16085 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/minicluster-data/master-0/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/minicluster-data/master-0/wal
uuid: "1970879255e148dfa0e77a084962857b"
format_stamp: "Formatted at 2025-09-03 08:08:05 on dist-test-slave-27r9"
server_key: "5514c2e176313fe6013060c9773f53a6"
server_key_iv: "f7da0afae38361edef328a24739e1a6b"
server_key_version: "encryptionkey@0"
I20250903 08:08:05.897090 16085 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:08:05.922835 16085 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:08:05.931279 16085 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:08:05.931471 16085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:08:05.938457 16085 rpc_server.cc:307] RPC server started. Bound to: 127.12.201.254:46375
I20250903 08:08:05.938458 16152 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.201.254:46375 every 8 connection(s)
I20250903 08:08:05.939529 16085 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/minicluster-data/master-0/data/info.pb
I20250903 08:08:05.941151 13095 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUML90r/build/debug/bin/kudu as pid 16085
I20250903 08:08:05.941274 13095 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/minicluster-data/master-0/wal/instance
I20250903 08:08:05.941527 13095 external_mini_cluster.cc:1468] Setting key 7f3ee8cb5c1b15cc2b1a4ae35d15798c
I20250903 08:08:05.942329 16153 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:08:05.948555 16153 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1970879255e148dfa0e77a084962857b: Bootstrap starting.
Sep 03 08:08:05 dist-test-slave-27r9 krb5kdc[16069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886885, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.201.254@KRBTEST.COM
I20250903 08:08:05.950933 16153 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1970879255e148dfa0e77a084962857b: Neither blocks nor log segments found. Creating new log.
I20250903 08:08:05.951787 16153 log.cc:826] T 00000000000000000000000000000000 P 1970879255e148dfa0e77a084962857b: Log is configured to *not* fsync() on all Append() calls
I20250903 08:08:05.953907 16153 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1970879255e148dfa0e77a084962857b: No bootstrap required, opened a new log
I20250903 08:08:05.954753 16156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:05.942727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33942 (local address 127.12.201.254:46375)
0903 08:08:05.943148 (+ 421us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:05.943159 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:05.943197 (+ 38us) server_negotiation.cc:408] Connection header received
0903 08:08:05.944095 (+ 898us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:05.944127 (+ 32us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:05.944525 (+ 398us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:05.944884 (+ 359us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:05.945626 (+ 742us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:05.946762 (+ 1136us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:05.947447 (+ 685us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:05.947719 (+ 272us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:05.950024 (+ 2305us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:05.950062 (+ 38us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:05.950073 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:08:05.950103 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:05.952133 (+ 2030us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:05.952632 (+ 499us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:05.952639 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:05.952644 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:05.952738 (+ 94us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:05.952996 (+ 258us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:05.952999 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:05.953001 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:05.953391 (+ 390us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:05.953599 (+ 208us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:05.953758 (+ 159us) server_negotiation.cc:300] Negotiation successful
0903 08:08:05.954029 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":123,"threads_started":1}
I20250903 08:08:05.956812 16153 raft_consensus.cc:357] T 00000000000000000000000000000000 P 1970879255e148dfa0e77a084962857b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1970879255e148dfa0e77a084962857b" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 46375 } }
I20250903 08:08:05.957000 16153 raft_consensus.cc:383] T 00000000000000000000000000000000 P 1970879255e148dfa0e77a084962857b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:08:05.957062 16153 raft_consensus.cc:738] T 00000000000000000000000000000000 P 1970879255e148dfa0e77a084962857b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1970879255e148dfa0e77a084962857b, State: Initialized, Role: FOLLOWER
I20250903 08:08:05.957417 16153 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1970879255e148dfa0e77a084962857b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1970879255e148dfa0e77a084962857b" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 46375 } }
I20250903 08:08:05.957526 16153 raft_consensus.cc:397] T 00000000000000000000000000000000 P 1970879255e148dfa0e77a084962857b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250903 08:08:05.957569 16153 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1970879255e148dfa0e77a084962857b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250903 08:08:05.957645 16153 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1970879255e148dfa0e77a084962857b [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:08:05.958403 16153 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1970879255e148dfa0e77a084962857b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1970879255e148dfa0e77a084962857b" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 46375 } }
I20250903 08:08:05.958683 16153 leader_election.cc:304] T 00000000000000000000000000000000 P 1970879255e148dfa0e77a084962857b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1970879255e148dfa0e77a084962857b; no voters:
I20250903 08:08:05.958911 16153 leader_election.cc:290] T 00000000000000000000000000000000 P 1970879255e148dfa0e77a084962857b [CANDIDATE]: Term 1 election: Requested vote from peers
I20250903 08:08:05.959057 16158 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 1970879255e148dfa0e77a084962857b [term 1 FOLLOWER]: Leader election won for term 1
I20250903 08:08:05.959252 16158 raft_consensus.cc:695] T 00000000000000000000000000000000 P 1970879255e148dfa0e77a084962857b [term 1 LEADER]: Becoming Leader. State: Replica: 1970879255e148dfa0e77a084962857b, State: Running, Role: LEADER
I20250903 08:08:05.959532 16158 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1970879255e148dfa0e77a084962857b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1970879255e148dfa0e77a084962857b" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 46375 } }
I20250903 08:08:05.959998 16153 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1970879255e148dfa0e77a084962857b [sys.catalog]: configured and running, proceeding with master startup.
I20250903 08:08:05.960997 16160 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1970879255e148dfa0e77a084962857b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1970879255e148dfa0e77a084962857b. Latest consensus state: current_term: 1 leader_uuid: "1970879255e148dfa0e77a084962857b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1970879255e148dfa0e77a084962857b" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 46375 } } }
I20250903 08:08:05.961133 16160 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1970879255e148dfa0e77a084962857b [sys.catalog]: This master's current role is: LEADER
I20250903 08:08:05.961371 16159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1970879255e148dfa0e77a084962857b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1970879255e148dfa0e77a084962857b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1970879255e148dfa0e77a084962857b" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 46375 } } }
I20250903 08:08:05.961463 16159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1970879255e148dfa0e77a084962857b [sys.catalog]: This master's current role is: LEADER
I20250903 08:08:05.961752 16162 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250903 08:08:05.964666 16162 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250903 08:08:05.970353 16162 catalog_manager.cc:1349] Generated new cluster ID: 0dd20f38a9e34f799cf4c9a3dae994f2
I20250903 08:08:05.970435 16162 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250903 08:08:05.981437 16162 catalog_manager.cc:1372] Generated new certificate authority record
I20250903 08:08:05.982295 16162 catalog_manager.cc:1506] Loading token signing keys...
I20250903 08:08:05.996762 16162 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 1970879255e148dfa0e77a084962857b: Generated new TSK 0
I20250903 08:08:05.997431 16162 catalog_manager.cc:1516] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.201.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.201.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.201.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.201.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.201.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.201.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.201.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.201.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/krb5kdc/kudu.keytab.
I20250903 08:08:06.056154 13095 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUML90r/build/debug/bin/kudu
/tmp/dist-test-taskUML90r/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.201.193:0
--local_ip_for_outbound_sockets=127.12.201.193
--webserver_interface=127.12.201.193
--webserver_port=0
--tserver_master_addrs=127.12.201.254:46375
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.201.212:39421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 08:08:06.163091 16181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:08:06.163308 16181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:08:06.163384 16181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:08:06.166507 16181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 08:08:06.166567 16181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:08:06.166671 16181 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.201.193
I20250903 08:08:06.170871 16181 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.201.212:39421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.201.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.201.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.12.201.254:46375
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16181
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.201.193
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 07:50:08 UTC on 5fd53c4cbb9d
build id 7874
I20250903 08:08:06.172065 16181 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:08:06.172977 16181 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:08:06.179575 16187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:06.179574 16186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:08:06.179858 16181 server_base.cc:1047] running on GCE node
W20250903 08:08:06.179601 16189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:08:06.180361 16181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:08:06.180914 16181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:08:06.182060 16181 hybrid_clock.cc:648] HybridClock initialized: now 1756886886182047 us; error 32 us; skew 500 ppm
Sep 03 08:08:06 dist-test-slave-27r9 krb5kdc[16069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886886, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 08:08:06.184881 16181 init.cc:377] Logged in from keytab as kudu/127.12.201.193@KRBTEST.COM (short username kudu)
I20250903 08:08:06.186023 16181 webserver.cc:480] Webserver started at http://127.12.201.193:36073/ using document root <none> and password file <none>
I20250903 08:08:06.186571 16181 fs_manager.cc:362] Metadata directory not provided
I20250903 08:08:06.186620 16181 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:08:06.186796 16181 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:08:06.188575 16181 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/minicluster-data/ts-0/data/instance:
uuid: "fbd55b212982445baa0397ccb601af7d"
format_stamp: "Formatted at 2025-09-03 08:08:06 on dist-test-slave-27r9"
server_key: "0e619e17aac799a13517d2473ea3c79d"
server_key_iv: "3c2e426cd244b480742c618e5246786d"
server_key_version: "encryptionkey@0"
I20250903 08:08:06.189031 16181 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/minicluster-data/ts-0/wal/instance:
uuid: "fbd55b212982445baa0397ccb601af7d"
format_stamp: "Formatted at 2025-09-03 08:08:06 on dist-test-slave-27r9"
server_key: "0e619e17aac799a13517d2473ea3c79d"
server_key_iv: "3c2e426cd244b480742c618e5246786d"
server_key_version: "encryptionkey@0"
I20250903 08:08:06.192340 16181 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250903 08:08:06.194459 16196 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:06.195465 16181 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250903 08:08:06.195562 16181 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/minicluster-data/ts-0/wal
uuid: "fbd55b212982445baa0397ccb601af7d"
format_stamp: "Formatted at 2025-09-03 08:08:06 on dist-test-slave-27r9"
server_key: "0e619e17aac799a13517d2473ea3c79d"
server_key_iv: "3c2e426cd244b480742c618e5246786d"
server_key_version: "encryptionkey@0"
I20250903 08:08:06.195649 16181 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:08:06.204942 16181 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:08:06.207595 16181 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:08:06.207759 16181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:08:06.208298 16181 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 08:08:06.209098 16181 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 08:08:06.209143 16181 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:06.209187 16181 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 08:08:06.209218 16181 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:06.217820 16181 rpc_server.cc:307] RPC server started. Bound to: 127.12.201.193:43473
I20250903 08:08:06.217842 16309 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.201.193:43473 every 8 connection(s)
I20250903 08:08:06.218737 16181 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/minicluster-data/ts-0/data/info.pb
I20250903 08:08:06.225237 13095 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUML90r/build/debug/bin/kudu as pid 16181
I20250903 08:08:06.225378 13095 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/minicluster-data/ts-0/wal/instance
I20250903 08:08:06.225626 13095 external_mini_cluster.cc:1468] Setting key 244bb43d80edb38b1f3df86d1489edb7
Sep 03 08:08:06 dist-test-slave-27r9 krb5kdc[16069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886886, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.193@KRBTEST.COM for kudu/127.12.201.254@KRBTEST.COM
I20250903 08:08:06.231252 16156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:06.220441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.193:45695 (local address 127.12.201.254:46375)
0903 08:08:06.220571 (+ 130us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:06.220575 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:06.221232 (+ 657us) server_negotiation.cc:408] Connection header received
0903 08:08:06.222025 (+ 793us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:06.222029 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:06.222083 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:06.222163 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:06.223890 (+ 1727us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:06.224393 (+ 503us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:06.225022 (+ 629us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:06.225244 (+ 222us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:06.227594 (+ 2350us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:06.227609 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:06.227611 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:08:06.227635 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:06.229105 (+ 1470us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:06.229697 (+ 592us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:06.229700 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:06.229701 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:06.229747 (+ 46us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:06.230056 (+ 309us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:06.230059 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:06.230060 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:06.230226 (+ 166us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:06.230310 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:06.230967 (+ 657us) server_negotiation.cc:300] Negotiation successful
0903 08:08:06.231113 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20250903 08:08:06.232272 16312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:06.220656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:46375 (local address 127.12.201.193:45695)
0903 08:08:06.221070 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:06.221106 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:06.221817 (+ 711us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:06.222315 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:06.222322 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:06.222702 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:08:06.223681 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:06.223696 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:06.224498 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:06.224500 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:06.224893 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:06.224899 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:06.225049 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:06.225698 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:08:06.225740 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:08:06.227447 (+ 1707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:08:06.229227 (+ 1780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:06.229235 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:06.229252 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:06.229588 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:06.229836 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:06.229842 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:06.229847 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:06.229968 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:06.230331 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:08:06.230341 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:08:06.230663 (+ 322us) client_negotiation.cc:769] Sending connection context
0903 08:08:06.230928 (+ 265us) client_negotiation.cc:240] Negotiation successful
0903 08:08:06.231282 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":107,"threads_started":1}
I20250903 08:08:06.233595 16310 heartbeater.cc:344] Connected to a master server at 127.12.201.254:46375
I20250903 08:08:06.233848 16310 heartbeater.cc:461] Registering TS with master...
I20250903 08:08:06.234397 16310 heartbeater.cc:507] Master 127.12.201.254:46375 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.12.201.194@KRBTEST.COM; defaulting to no policy
I20250903 08:08:06.235881 16117 ts_manager.cc:194] Registered new tserver with Master: fbd55b212982445baa0397ccb601af7d (127.12.201.193:43473)
I20250903 08:08:06.237069 16117 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.201.193@KRBTEST.COM'} at 127.12.201.193:45695
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.201.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.201.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.201.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.201.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.201.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.201.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.201.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/krb5kdc/kudu.keytab.
I20250903 08:08:06.275235 13095 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUML90r/build/debug/bin/kudu
/tmp/dist-test-taskUML90r/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.201.194:0
--local_ip_for_outbound_sockets=127.12.201.194
--webserver_interface=127.12.201.194
--webserver_port=0
--tserver_master_addrs=127.12.201.254:46375
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.201.212:39421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 08:08:06.380717 16317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:08:06.380930 16317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:08:06.380988 16317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:08:06.384173 16317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 08:08:06.384230 16317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:08:06.384307 16317 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.201.194
I20250903 08:08:06.388381 16317 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.201.212:39421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.201.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.201.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.12.201.254:46375
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16317
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.201.194
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 07:50:08 UTC on 5fd53c4cbb9d
build id 7874
I20250903 08:08:06.389323 16317 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:08:06.390141 16317 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:08:06.396106 16322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:06.396132 16323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:06.396112 16325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:08:06.396387 16317 server_base.cc:1047] running on GCE node
I20250903 08:08:06.396726 16317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:08:06.397372 16317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:08:06.398525 16317 hybrid_clock.cc:648] HybridClock initialized: now 1756886886398506 us; error 42 us; skew 500 ppm
Sep 03 08:08:06 dist-test-slave-27r9 krb5kdc[16069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886886, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 08:08:06.401165 16317 init.cc:377] Logged in from keytab as kudu/127.12.201.194@KRBTEST.COM (short username kudu)
I20250903 08:08:06.402140 16317 webserver.cc:480] Webserver started at http://127.12.201.194:36163/ using document root <none> and password file <none>
I20250903 08:08:06.402678 16317 fs_manager.cc:362] Metadata directory not provided
I20250903 08:08:06.402724 16317 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:08:06.402901 16317 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:08:06.404595 16317 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/minicluster-data/ts-1/data/instance:
uuid: "854c99247a54448e86520a6c3d902594"
format_stamp: "Formatted at 2025-09-03 08:08:06 on dist-test-slave-27r9"
server_key: "f6e54bcce145dde32899a542b21206de"
server_key_iv: "2928697d22f58e967167ebaca8d11882"
server_key_version: "encryptionkey@0"
I20250903 08:08:06.405062 16317 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/minicluster-data/ts-1/wal/instance:
uuid: "854c99247a54448e86520a6c3d902594"
format_stamp: "Formatted at 2025-09-03 08:08:06 on dist-test-slave-27r9"
server_key: "f6e54bcce145dde32899a542b21206de"
server_key_iv: "2928697d22f58e967167ebaca8d11882"
server_key_version: "encryptionkey@0"
I20250903 08:08:06.408394 16317 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20250903 08:08:06.410465 16332 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:06.411449 16317 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250903 08:08:06.411540 16317 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/minicluster-data/ts-1/wal
uuid: "854c99247a54448e86520a6c3d902594"
format_stamp: "Formatted at 2025-09-03 08:08:06 on dist-test-slave-27r9"
server_key: "f6e54bcce145dde32899a542b21206de"
server_key_iv: "2928697d22f58e967167ebaca8d11882"
server_key_version: "encryptionkey@0"
I20250903 08:08:06.411635 16317 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:08:06.424686 16317 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:08:06.427258 16317 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:08:06.427426 16317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:08:06.427954 16317 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 08:08:06.428736 16317 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 08:08:06.428781 16317 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:06.428834 16317 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 08:08:06.428866 16317 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:06.437692 16317 rpc_server.cc:307] RPC server started. Bound to: 127.12.201.194:40395
I20250903 08:08:06.437727 16445 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.201.194:40395 every 8 connection(s)
I20250903 08:08:06.438588 16317 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/minicluster-data/ts-1/data/info.pb
I20250903 08:08:06.440940 13095 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUML90r/build/debug/bin/kudu as pid 16317
I20250903 08:08:06.441064 13095 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/minicluster-data/ts-1/wal/instance
I20250903 08:08:06.441288 13095 external_mini_cluster.cc:1468] Setting key dccf61e6cb6ff7c902b38f6898382cf4
Sep 03 08:08:06 dist-test-slave-27r9 krb5kdc[16069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886886, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.194@KRBTEST.COM for kudu/127.12.201.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.201.195@KRBTEST.COM; defaulting to no policy
I20250903 08:08:06.451333 16156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:06.440204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.194:57777 (local address 127.12.201.254:46375)
0903 08:08:06.440327 (+ 123us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:06.440331 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:06.440925 (+ 594us) server_negotiation.cc:408] Connection header received
0903 08:08:06.441819 (+ 894us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:06.441823 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:06.441879 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:06.441973 (+ 94us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:06.443803 (+ 1830us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:06.444528 (+ 725us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:06.445430 (+ 902us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:06.445556 (+ 126us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:06.448004 (+ 2448us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:06.448020 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:06.448022 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:08:06.448052 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:06.449470 (+ 1418us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:06.449939 (+ 469us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:06.449942 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:06.449944 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:06.449994 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:06.450262 (+ 268us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:06.450265 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:06.450267 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:06.450424 (+ 157us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:06.450513 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:06.451034 (+ 521us) server_negotiation.cc:300] Negotiation successful
0903 08:08:06.451166 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20250903 08:08:06.451992 16448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:06.440397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:46375 (local address 127.12.201.194:57777)
0903 08:08:06.440779 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:06.440812 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:06.441621 (+ 809us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:06.442144 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:06.442154 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:06.442618 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:08:06.443644 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:06.443659 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:06.444643 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:06.444650 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:06.445318 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:06.445332 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:06.445535 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:06.446171 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:08:06.446209 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:08:06.447855 (+ 1646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:08:06.449588 (+ 1733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:06.449594 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:06.449605 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:06.449841 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:06.450086 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:06.450089 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:06.450093 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:06.450175 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:06.450527 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:08:06.450535 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:08:06.450754 (+ 219us) client_negotiation.cc:769] Sending connection context
0903 08:08:06.450966 (+ 212us) client_negotiation.cc:240] Negotiation successful
0903 08:08:06.451254 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":101,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.201.195@KRBTEST.COM" created.
I20250903 08:08:06.453203 16446 heartbeater.cc:344] Connected to a master server at 127.12.201.254:46375
I20250903 08:08:06.453434 16446 heartbeater.cc:461] Registering TS with master...
I20250903 08:08:06.454020 16446 heartbeater.cc:507] Master 127.12.201.254:46375 requested a full tablet report, sending...
I20250903 08:08:06.455189 16117 ts_manager.cc:194] Registered new tserver with Master: 854c99247a54448e86520a6c3d902594 (127.12.201.194:40395)
I20250903 08:08:06.455772 16117 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.201.194@KRBTEST.COM'} at 127.12.201.194:57777
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.201.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.201.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.201.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.201.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.201.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.201.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/krb5kdc/kudu.keytab.
I20250903 08:08:06.486346 13095 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUML90r/build/debug/bin/kudu
/tmp/dist-test-taskUML90r/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.201.195:0
--local_ip_for_outbound_sockets=127.12.201.195
--webserver_interface=127.12.201.195
--webserver_port=0
--tserver_master_addrs=127.12.201.254:46375
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.201.212:39421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 08:08:06.590662 16453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:08:06.590893 16453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:08:06.590950 16453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:08:06.594099 16453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 08:08:06.594166 16453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:08:06.594244 16453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.201.195
I20250903 08:08:06.598346 16453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.201.212:39421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.201.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.201.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.12.201.254:46375
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16453
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.201.195
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 07:50:08 UTC on 5fd53c4cbb9d
build id 7874
I20250903 08:08:06.599422 16453 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:08:06.600273 16453 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:08:06.606592 16461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:06.606595 16459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:06.606595 16458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:08:06.606704 16453 server_base.cc:1047] running on GCE node
I20250903 08:08:06.607234 16453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:08:06.607733 16453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:08:06.608888 16453 hybrid_clock.cc:648] HybridClock initialized: now 1756886886608880 us; error 31 us; skew 500 ppm
Sep 03 08:08:06 dist-test-slave-27r9 krb5kdc[16069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886886, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 08:08:06.611430 16453 init.cc:377] Logged in from keytab as kudu/127.12.201.195@KRBTEST.COM (short username kudu)
I20250903 08:08:06.612393 16453 webserver.cc:480] Webserver started at http://127.12.201.195:35959/ using document root <none> and password file <none>
I20250903 08:08:06.612917 16453 fs_manager.cc:362] Metadata directory not provided
I20250903 08:08:06.612968 16453 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:08:06.613164 16453 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:08:06.614876 16453 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/minicluster-data/ts-2/data/instance:
uuid: "150cccb437994780b8c3b6e1de4ee629"
format_stamp: "Formatted at 2025-09-03 08:08:06 on dist-test-slave-27r9"
server_key: "a7c61abafe8e3857d792bbd1ee88d634"
server_key_iv: "4be968bf12db60283dfd0aa7f5057c39"
server_key_version: "encryptionkey@0"
I20250903 08:08:06.615433 16453 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/minicluster-data/ts-2/wal/instance:
uuid: "150cccb437994780b8c3b6e1de4ee629"
format_stamp: "Formatted at 2025-09-03 08:08:06 on dist-test-slave-27r9"
server_key: "a7c61abafe8e3857d792bbd1ee88d634"
server_key_iv: "4be968bf12db60283dfd0aa7f5057c39"
server_key_version: "encryptionkey@0"
I20250903 08:08:06.618678 16453 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20250903 08:08:06.620894 16468 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:06.621845 16453 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250903 08:08:06.621937 16453 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/minicluster-data/ts-2/wal
uuid: "150cccb437994780b8c3b6e1de4ee629"
format_stamp: "Formatted at 2025-09-03 08:08:06 on dist-test-slave-27r9"
server_key: "a7c61abafe8e3857d792bbd1ee88d634"
server_key_iv: "4be968bf12db60283dfd0aa7f5057c39"
server_key_version: "encryptionkey@0"
I20250903 08:08:06.622030 16453 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:08:06.640807 16453 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:08:06.643386 16453 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:08:06.643553 16453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:08:06.644059 16453 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 08:08:06.644836 16453 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 08:08:06.644882 16453 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:06.644924 16453 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 08:08:06.644960 16453 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:06.653457 16453 rpc_server.cc:307] RPC server started. Bound to: 127.12.201.195:33515
I20250903 08:08:06.653491 16581 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.201.195:33515 every 8 connection(s)
I20250903 08:08:06.654378 16453 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/minicluster-data/ts-2/data/info.pb
I20250903 08:08:06.661738 13095 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUML90r/build/debug/bin/kudu as pid 16453
I20250903 08:08:06.661868 13095 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756886860253814-13095-0/minicluster-data/ts-2/wal/instance
I20250903 08:08:06.662106 13095 external_mini_cluster.cc:1468] Setting key 8dec3090d4a4127dfdb891fbc4a2fc1e
Sep 03 08:08:06 dist-test-slave-27r9 krb5kdc[16069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886886, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.195@KRBTEST.COM for kudu/127.12.201.254@KRBTEST.COM
I20250903 08:08:06.665967 16156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:06.656074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.195:46117 (local address 127.12.201.254:46375)
0903 08:08:06.656216 (+ 142us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:06.656219 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:06.656902 (+ 683us) server_negotiation.cc:408] Connection header received
0903 08:08:06.657705 (+ 803us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:06.657709 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:06.657769 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:06.657848 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:06.659457 (+ 1609us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:06.659946 (+ 489us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:06.660552 (+ 606us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:06.660723 (+ 171us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:06.663032 (+ 2309us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:06.663053 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:06.663058 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:08:06.663089 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:06.664546 (+ 1457us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:06.664952 (+ 406us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:06.664955 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:06.664956 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:06.664993 (+ 37us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:06.665202 (+ 209us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:06.665204 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:06.665205 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:06.665326 (+ 121us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:06.665392 (+ 66us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:06.665762 (+ 370us) server_negotiation.cc:300] Negotiation successful
0903 08:08:06.665853 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20250903 08:08:06.666538 16584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:06.656315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:46375 (local address 127.12.201.195:46117)
0903 08:08:06.656749 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:06.656784 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:06.657523 (+ 739us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:06.658009 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:06.658017 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:06.658378 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:08:06.659315 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:06.659330 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:06.660068 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:06.660072 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:06.660447 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:06.660454 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:06.660631 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:06.661229 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:08:06.661250 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:08:06.662853 (+ 1603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:08:06.664644 (+ 1791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:06.664648 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:06.664656 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:06.664875 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:06.665055 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:06.665056 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:06.665058 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:06.665140 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:06.665392 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:08:06.665396 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:08:06.665587 (+ 191us) client_negotiation.cc:769] Sending connection context
0903 08:08:06.665738 (+ 151us) client_negotiation.cc:240] Negotiation successful
0903 08:08:06.665928 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":106,"threads_started":1}
I20250903 08:08:06.667470 16582 heartbeater.cc:344] Connected to a master server at 127.12.201.254:46375
I20250903 08:08:06.667678 16582 heartbeater.cc:461] Registering TS with master...
I20250903 08:08:06.668143 16582 heartbeater.cc:507] Master 127.12.201.254:46375 requested a full tablet report, sending...
I20250903 08:08:06.669006 16117 ts_manager.cc:194] Registered new tserver with Master: 150cccb437994780b8c3b6e1de4ee629 (127.12.201.195:33515)
I20250903 08:08:06.669518 16117 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.201.195@KRBTEST.COM'} at 127.12.201.195:46117
I20250903 08:08:06.675902 13095 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20250903 08:08:06.799994 16156 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 08:08:06.796537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33948 (local address 127.12.201.254:46375)
0903 08:08:06.796678 (+ 141us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:06.796682 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:06.797550 (+ 868us) server_negotiation.cc:408] Connection header received
0903 08:08:06.798439 (+ 889us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:06.798443 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:06.798505 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:06.798583 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:06.799860 (+ 1277us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33948: BlockingRecv error: recv got EOF from 127.0.0.1:33948 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20250903 08:08:06.801386 16156 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 08:08:06.800560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33958 (local address 127.12.201.254:46375)
0903 08:08:06.800674 (+ 114us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:06.800678 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:06.800692 (+ 14us) server_negotiation.cc:408] Connection header received
0903 08:08:06.800788 (+ 96us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:06.800794 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:06.800850 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:06.800948 (+ 98us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:06.801278 (+ 330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33958: BlockingRecv error: recv got EOF from 127.0.0.1:33958 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20250903 08:08:06.802434 16156 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 08:08:06.801734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33964 (local address 127.12.201.254:46375)
0903 08:08:06.801830 (+ 96us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:06.801834 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:06.801847 (+ 13us) server_negotiation.cc:408] Connection header received
0903 08:08:06.801879 (+ 32us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:06.801881 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:06.801911 (+ 30us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:06.801973 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:06.802337 (+ 364us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33964: BlockingRecv error: recv got EOF from 127.0.0.1:33964 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20250903 08:08:06.806604 16156 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 08:08:06.805757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33978 (local address 127.12.201.254:46375)
0903 08:08:06.805867 (+ 110us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:06.805870 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:06.805949 (+ 79us) server_negotiation.cc:408] Connection header received
0903 08:08:06.806077 (+ 128us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:06.806079 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:06.806114 (+ 35us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:06.806186 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:06.806541 (+ 355us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33978: BlockingRecv error: recv got EOF from 127.0.0.1:33978 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
I20250903 08:08:06.814786 13095 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUML90r/build/debug/bin/kudu with pid 16181
I20250903 08:08:06.821925 13095 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUML90r/build/debug/bin/kudu with pid 16317
I20250903 08:08:06.827179 13095 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUML90r/build/debug/bin/kudu with pid 16453
I20250903 08:08:06.832459 13095 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUML90r/build/debug/bin/kudu with pid 16085
2025-09-03T08:08:06Z chronyd exiting
[ OK ] SecurityITest.TestRebalanceReportUnauthorized (3172 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1756886860253814-13095-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 03 08:08:06 dist-test-slave-27r9 krb5kdc[16605](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 03 08:08:06 dist-test-slave-27r9 krb5kdc[16605](info): set up 2 sockets
Sep 03 08:08:06 dist-test-slave-27r9 krb5kdc[16605](info): commencing operation
krb5kdc: starting...
W20250903 08:08:08.815318 13095 mini_kdc.cc:121] Time spent starting KDC: real 1.961s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1756886860253814-13095-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1756886860253814-13095-0/krb5kdc/test-user.keytab.
Sep 03 08:08:08 dist-test-slave-27r9 krb5kdc[16605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886888, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-09-03T08:08:08Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-03T08:08:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.201.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.201.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.201.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.201.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1756886860253814-13095-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.201.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.201.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.201.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.201.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1756886860253814-13095-0/krb5kdc/kudu.keytab.
I20250903 08:08:08.943704 13095 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUML90r/build/debug/bin/kudu
/tmp/dist-test-taskUML90r/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1756886860253814-13095-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1756886860253814-13095-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1756886860253814-13095-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1756886860253814-13095-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.201.254:40427
--webserver_interface=127.12.201.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.201.212:32819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.201.254:40427
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1756886860253814-13095-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1756886860253814-13095-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1756886860253814-13095-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 08:08:09.047362 16621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:08:09.047583 16621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:08:09.047643 16621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:08:09.050849 16621 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250903 08:08:09.050908 16621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 08:08:09.050931 16621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:08:09.050949 16621 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250903 08:08:09.050971 16621 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250903 08:08:09.055161 16621 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.201.212:32819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1756886860253814-13095-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1756886860253814-13095-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.201.254:40427
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.201.254:40427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1756886860253814-13095-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.201.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16621
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1756886860253814-13095-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 07:50:08 UTC on 5fd53c4cbb9d
build id 7874
I20250903 08:08:09.056195 16621 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:08:09.057061 16621 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:08:09.062517 16627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:09.062532 16626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:09.062525 16629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:08:09.062830 16621 server_base.cc:1047] running on GCE node
I20250903 08:08:09.063334 16621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:08:09.064216 16621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:08:09.065358 16621 hybrid_clock.cc:648] HybridClock initialized: now 1756886889065334 us; error 37 us; skew 500 ppm
Sep 03 08:08:09 dist-test-slave-27r9 krb5kdc[16605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886889, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 08:08:09.068073 16621 init.cc:377] Logged in from keytab as kudu/127.12.201.254@KRBTEST.COM (short username kudu)
I20250903 08:08:09.069099 16621 webserver.cc:480] Webserver started at http://127.12.201.254:44359/ using document root <none> and password file <none>
I20250903 08:08:09.069614 16621 fs_manager.cc:362] Metadata directory not provided
I20250903 08:08:09.069662 16621 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:08:09.069839 16621 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:08:09.071542 16621 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1756886860253814-13095-0/minicluster-data/master-0/data/instance:
uuid: "2b6c9418831244c598009417feb9ecd4"
format_stamp: "Formatted at 2025-09-03 08:08:09 on dist-test-slave-27r9"
server_key: "261c282739ce8d201f72b4f089405a9b"
server_key_iv: "d427f99667e8ae1c526740df1eccc725"
server_key_version: "encryptionkey@0"
I20250903 08:08:09.072000 16621 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1756886860253814-13095-0/minicluster-data/master-0/wal/instance:
uuid: "2b6c9418831244c598009417feb9ecd4"
format_stamp: "Formatted at 2025-09-03 08:08:09 on dist-test-slave-27r9"
server_key: "261c282739ce8d201f72b4f089405a9b"
server_key_iv: "d427f99667e8ae1c526740df1eccc725"
server_key_version: "encryptionkey@0"
I20250903 08:08:09.075244 16621 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250903 08:08:09.077203 16636 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:09.078131 16621 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250903 08:08:09.078238 16621 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1756886860253814-13095-0/minicluster-data/master-0/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1756886860253814-13095-0/minicluster-data/master-0/wal
uuid: "2b6c9418831244c598009417feb9ecd4"
format_stamp: "Formatted at 2025-09-03 08:08:09 on dist-test-slave-27r9"
server_key: "261c282739ce8d201f72b4f089405a9b"
server_key_iv: "d427f99667e8ae1c526740df1eccc725"
server_key_version: "encryptionkey@0"
I20250903 08:08:09.078318 16621 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1756886860253814-13095-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1756886860253814-13095-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1756886860253814-13095-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:08:09.089787 16621 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:08:09.092283 16621 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:08:09.092438 16621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:08:09.098935 16621 rpc_server.cc:307] RPC server started. Bound to: 127.12.201.254:40427
I20250903 08:08:09.099012 16688 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.201.254:40427 every 8 connection(s)
I20250903 08:08:09.099898 16621 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1756886860253814-13095-0/minicluster-data/master-0/data/info.pb
I20250903 08:08:09.102187 16689 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:08:09.107323 16689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b6c9418831244c598009417feb9ecd4: Bootstrap starting.
I20250903 08:08:09.109082 13095 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUML90r/build/debug/bin/kudu as pid 16621
I20250903 08:08:09.109198 13095 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1756886860253814-13095-0/minicluster-data/master-0/wal/instance
I20250903 08:08:09.109421 13095 external_mini_cluster.cc:1468] Setting key 0c36020d13e4a70a35589edaa36a70b1
I20250903 08:08:09.109709 16689 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b6c9418831244c598009417feb9ecd4: Neither blocks nor log segments found. Creating new log.
I20250903 08:08:09.110579 16689 log.cc:826] T 00000000000000000000000000000000 P 2b6c9418831244c598009417feb9ecd4: Log is configured to *not* fsync() on all Append() calls
I20250903 08:08:09.112438 16689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b6c9418831244c598009417feb9ecd4: No bootstrap required, opened a new log
I20250903 08:08:09.115269 16689 raft_consensus.cc:357] T 00000000000000000000000000000000 P 2b6c9418831244c598009417feb9ecd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b6c9418831244c598009417feb9ecd4" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 40427 } }
I20250903 08:08:09.115497 16689 raft_consensus.cc:383] T 00000000000000000000000000000000 P 2b6c9418831244c598009417feb9ecd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:08:09.115562 16689 raft_consensus.cc:738] T 00000000000000000000000000000000 P 2b6c9418831244c598009417feb9ecd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b6c9418831244c598009417feb9ecd4, State: Initialized, Role: FOLLOWER
I20250903 08:08:09.116079 16689 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b6c9418831244c598009417feb9ecd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b6c9418831244c598009417feb9ecd4" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 40427 } }
I20250903 08:08:09.116248 16689 raft_consensus.cc:397] T 00000000000000000000000000000000 P 2b6c9418831244c598009417feb9ecd4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250903 08:08:09.116326 16689 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2b6c9418831244c598009417feb9ecd4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250903 08:08:09.116434 16689 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 2b6c9418831244c598009417feb9ecd4 [term 0 FOLLOWER]: Advancing to term 1
Sep 03 08:08:09 dist-test-slave-27r9 krb5kdc[16605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886888, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.201.254@KRBTEST.COM
I20250903 08:08:09.117274 16689 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2b6c9418831244c598009417feb9ecd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b6c9418831244c598009417feb9ecd4" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 40427 } }
I20250903 08:08:09.117578 16689 leader_election.cc:304] T 00000000000000000000000000000000 P 2b6c9418831244c598009417feb9ecd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b6c9418831244c598009417feb9ecd4; no voters:
I20250903 08:08:09.117820 16689 leader_election.cc:290] T 00000000000000000000000000000000 P 2b6c9418831244c598009417feb9ecd4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250903 08:08:09.117933 16694 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 2b6c9418831244c598009417feb9ecd4 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 08:08:09.118134 16694 raft_consensus.cc:695] T 00000000000000000000000000000000 P 2b6c9418831244c598009417feb9ecd4 [term 1 LEADER]: Becoming Leader. State: Replica: 2b6c9418831244c598009417feb9ecd4, State: Running, Role: LEADER
I20250903 08:08:09.118445 16694 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b6c9418831244c598009417feb9ecd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b6c9418831244c598009417feb9ecd4" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 40427 } }
I20250903 08:08:09.119052 16689 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b6c9418831244c598009417feb9ecd4 [sys.catalog]: configured and running, proceeding with master startup.
I20250903 08:08:09.119566 16696 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b6c9418831244c598009417feb9ecd4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b6c9418831244c598009417feb9ecd4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b6c9418831244c598009417feb9ecd4" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 40427 } } }
I20250903 08:08:09.119684 16696 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b6c9418831244c598009417feb9ecd4 [sys.catalog]: This master's current role is: LEADER
I20250903 08:08:09.119920 16696 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b6c9418831244c598009417feb9ecd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b6c9418831244c598009417feb9ecd4. Latest consensus state: current_term: 1 leader_uuid: "2b6c9418831244c598009417feb9ecd4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b6c9418831244c598009417feb9ecd4" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 40427 } } }
I20250903 08:08:09.120007 16696 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b6c9418831244c598009417feb9ecd4 [sys.catalog]: This master's current role is: LEADER
I20250903 08:08:09.120306 16699 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250903 08:08:09.123323 16699 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250903 08:08:09.123531 16692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:09.110395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42404 (local address 127.12.201.254:40427)
0903 08:08:09.110762 (+ 367us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:09.110771 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:09.110812 (+ 41us) server_negotiation.cc:408] Connection header received
0903 08:08:09.111497 (+ 685us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:09.111518 (+ 21us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:09.111794 (+ 276us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:09.112063 (+ 269us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:09.113062 (+ 999us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:09.113783 (+ 721us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:09.114618 (+ 835us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:09.114819 (+ 201us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:09.117320 (+ 2501us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:09.117338 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:09.117347 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:08:09.117379 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:09.120274 (+ 2895us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:09.120732 (+ 458us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:09.120740 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:09.120747 (+ 7us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:09.120864 (+ 117us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:09.121146 (+ 282us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:09.121150 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:09.121152 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:09.121562 (+ 410us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:09.121729 (+ 167us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:09.122516 (+ 787us) server_negotiation.cc:300] Negotiation successful
0903 08:08:09.122770 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":110,"threads_started":1}
I20250903 08:08:09.126246 13095 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20250903 08:08:09.129004 16699 catalog_manager.cc:1349] Generated new cluster ID: 3b9e25c556e84c39ab8dfcaa4a2453bb
I20250903 08:08:09.129073 16699 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250903 08:08:09.152884 16692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:09.148190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42418 (local address 127.12.201.254:40427)
0903 08:08:09.148313 (+ 123us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:09.148317 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:09.148501 (+ 184us) server_negotiation.cc:408] Connection header received
0903 08:08:09.148629 (+ 128us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:09.148632 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:09.148681 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:09.148771 (+ 90us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:09.149531 (+ 760us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:09.150259 (+ 728us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:09.151082 (+ 823us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:09.151257 (+ 175us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:09.151857 (+ 600us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:09.151877 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:09.151880 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0903 08:08:09.152433 (+ 553us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:09.152463 (+ 30us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0903 08:08:09.152480 (+ 17us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:09.152565 (+ 85us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:09.152671 (+ 106us) server_negotiation.cc:300] Negotiation successful
0903 08:08:09.152725 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20250903 08:08:09.156510 16699 catalog_manager.cc:1372] Generated new certificate authority record
I20250903 08:08:09.157678 16699 catalog_manager.cc:1506] Loading token signing keys...
I20250903 08:08:09.167542 16699 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 2b6c9418831244c598009417feb9ecd4: Generated new TSK 0
I20250903 08:08:09.168377 16699 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250903 08:08:09.174823 13095 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUML90r/build/debug/bin/kudu with pid 16621
2025-09-03T08:08:09Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2339 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 03 08:08:09 dist-test-slave-27r9 krb5kdc[16722](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 03 08:08:09 dist-test-slave-27r9 krb5kdc[16722](info): set up 2 sockets
Sep 03 08:08:09 dist-test-slave-27r9 krb5kdc[16722](info): commencing operation
krb5kdc: starting...
W20250903 08:08:11.143379 13095 mini_kdc.cc:121] Time spent starting KDC: real 1.950s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/krb5kdc/test-user.keytab.
Sep 03 08:08:11 dist-test-slave-27r9 krb5kdc[16722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886891, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-09-03T08:08:11Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-03T08:08:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.201.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.201.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.201.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.201.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.201.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.201.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.201.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.201.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/krb5kdc/kudu.keytab.
I20250903 08:08:11.267997 13095 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUML90r/build/debug/bin/kudu
/tmp/dist-test-taskUML90r/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.201.254:43079
--webserver_interface=127.12.201.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.201.212:42945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.201.254:43079
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 08:08:11.367233 16738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:08:11.367408 16738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:08:11.367455 16738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:08:11.370365 16738 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250903 08:08:11.370409 16738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 08:08:11.370429 16738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:08:11.370465 16738 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250903 08:08:11.370481 16738 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250903 08:08:11.374356 16738 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.201.212:42945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.201.254:43079
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.201.254:43079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.201.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16738
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 07:50:08 UTC on 5fd53c4cbb9d
build id 7874
I20250903 08:08:11.375242 16738 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:08:11.375988 16738 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:08:11.381040 16744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:08:11.381150 16738 server_base.cc:1047] running on GCE node
W20250903 08:08:11.381012 16746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:11.381055 16743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:08:11.381690 16738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:08:11.382584 16738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:08:11.383737 16738 hybrid_clock.cc:648] HybridClock initialized: now 1756886891383707 us; error 42 us; skew 500 ppm
Sep 03 08:08:11 dist-test-slave-27r9 krb5kdc[16722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886891, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 08:08:11.386036 16738 init.cc:377] Logged in from keytab as kudu/127.12.201.254@KRBTEST.COM (short username kudu)
I20250903 08:08:11.386992 16738 webserver.cc:480] Webserver started at http://127.12.201.254:46409/ using document root <none> and password file <none>
I20250903 08:08:11.387512 16738 fs_manager.cc:362] Metadata directory not provided
I20250903 08:08:11.387568 16738 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:08:11.387773 16738 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:08:11.389402 16738 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/minicluster-data/master-0/data/instance:
uuid: "8f8994423f61427b84bb24698530b26b"
format_stamp: "Formatted at 2025-09-03 08:08:11 on dist-test-slave-27r9"
server_key: "2113042faffb5bde75c6487e192f11c2"
server_key_iv: "d6d246f67ce8810f13601261eaf8593f"
server_key_version: "encryptionkey@0"
I20250903 08:08:11.389864 16738 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/minicluster-data/master-0/wal/instance:
uuid: "8f8994423f61427b84bb24698530b26b"
format_stamp: "Formatted at 2025-09-03 08:08:11 on dist-test-slave-27r9"
server_key: "2113042faffb5bde75c6487e192f11c2"
server_key_iv: "d6d246f67ce8810f13601261eaf8593f"
server_key_version: "encryptionkey@0"
I20250903 08:08:11.393023 16738 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250903 08:08:11.395074 16753 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:11.396030 16738 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250903 08:08:11.396136 16738 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/minicluster-data/master-0/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/minicluster-data/master-0/wal
uuid: "8f8994423f61427b84bb24698530b26b"
format_stamp: "Formatted at 2025-09-03 08:08:11 on dist-test-slave-27r9"
server_key: "2113042faffb5bde75c6487e192f11c2"
server_key_iv: "d6d246f67ce8810f13601261eaf8593f"
server_key_version: "encryptionkey@0"
I20250903 08:08:11.396219 16738 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:08:11.415153 16738 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:08:11.423112 16738 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:08:11.423321 16738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:08:11.430451 16805 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.201.254:43079 every 8 connection(s)
I20250903 08:08:11.430459 16738 rpc_server.cc:307] RPC server started. Bound to: 127.12.201.254:43079
I20250903 08:08:11.431519 16738 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/minicluster-data/master-0/data/info.pb
I20250903 08:08:11.433159 13095 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUML90r/build/debug/bin/kudu as pid 16738
I20250903 08:08:11.433291 13095 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/minicluster-data/master-0/wal/instance
I20250903 08:08:11.433514 13095 external_mini_cluster.cc:1468] Setting key 0b392e0585d171f45fec625433053be8
I20250903 08:08:11.434101 16806 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:08:11.440538 16806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f8994423f61427b84bb24698530b26b: Bootstrap starting.
Sep 03 08:08:11 dist-test-slave-27r9 krb5kdc[16722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886891, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.201.254@KRBTEST.COM
I20250903 08:08:11.442936 16806 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f8994423f61427b84bb24698530b26b: Neither blocks nor log segments found. Creating new log.
I20250903 08:08:11.443821 16806 log.cc:826] T 00000000000000000000000000000000 P 8f8994423f61427b84bb24698530b26b: Log is configured to *not* fsync() on all Append() calls
I20250903 08:08:11.445785 16806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f8994423f61427b84bb24698530b26b: No bootstrap required, opened a new log
I20250903 08:08:11.447047 16809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:11.434756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33542 (local address 127.12.201.254:43079)
0903 08:08:11.435167 (+ 411us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:11.435178 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:11.435217 (+ 39us) server_negotiation.cc:408] Connection header received
0903 08:08:11.436485 (+ 1268us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:11.436512 (+ 27us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:11.436868 (+ 356us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:11.437225 (+ 357us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:11.438059 (+ 834us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:11.439234 (+ 1175us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:11.439849 (+ 615us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:11.440106 (+ 257us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:11.442344 (+ 2238us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:11.442358 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:11.442370 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:08:11.442407 (+ 37us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:11.444555 (+ 2148us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:11.444953 (+ 398us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:11.444959 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:11.444965 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:11.445055 (+ 90us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:11.445243 (+ 188us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:11.445246 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:11.445247 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:11.445635 (+ 388us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:11.445803 (+ 168us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:11.446042 (+ 239us) server_negotiation.cc:300] Negotiation successful
0903 08:08:11.446310 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":105,"threads_started":1}
I20250903 08:08:11.448710 16806 raft_consensus.cc:357] T 00000000000000000000000000000000 P 8f8994423f61427b84bb24698530b26b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f8994423f61427b84bb24698530b26b" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 43079 } }
I20250903 08:08:11.448917 16806 raft_consensus.cc:383] T 00000000000000000000000000000000 P 8f8994423f61427b84bb24698530b26b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:08:11.449018 16806 raft_consensus.cc:738] T 00000000000000000000000000000000 P 8f8994423f61427b84bb24698530b26b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f8994423f61427b84bb24698530b26b, State: Initialized, Role: FOLLOWER
I20250903 08:08:11.449417 16806 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f8994423f61427b84bb24698530b26b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f8994423f61427b84bb24698530b26b" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 43079 } }
I20250903 08:08:11.449534 16806 raft_consensus.cc:397] T 00000000000000000000000000000000 P 8f8994423f61427b84bb24698530b26b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250903 08:08:11.449594 16806 raft_consensus.cc:491] T 00000000000000000000000000000000 P 8f8994423f61427b84bb24698530b26b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250903 08:08:11.449695 16806 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 8f8994423f61427b84bb24698530b26b [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:08:11.450488 16806 raft_consensus.cc:513] T 00000000000000000000000000000000 P 8f8994423f61427b84bb24698530b26b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f8994423f61427b84bb24698530b26b" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 43079 } }
I20250903 08:08:11.450793 16806 leader_election.cc:304] T 00000000000000000000000000000000 P 8f8994423f61427b84bb24698530b26b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f8994423f61427b84bb24698530b26b; no voters:
I20250903 08:08:11.451077 16806 leader_election.cc:290] T 00000000000000000000000000000000 P 8f8994423f61427b84bb24698530b26b [CANDIDATE]: Term 1 election: Requested vote from peers
I20250903 08:08:11.451187 16811 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 8f8994423f61427b84bb24698530b26b [term 1 FOLLOWER]: Leader election won for term 1
I20250903 08:08:11.451382 16811 raft_consensus.cc:695] T 00000000000000000000000000000000 P 8f8994423f61427b84bb24698530b26b [term 1 LEADER]: Becoming Leader. State: Replica: 8f8994423f61427b84bb24698530b26b, State: Running, Role: LEADER
I20250903 08:08:11.451699 16811 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f8994423f61427b84bb24698530b26b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f8994423f61427b84bb24698530b26b" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 43079 } }
I20250903 08:08:11.452047 16806 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f8994423f61427b84bb24698530b26b [sys.catalog]: configured and running, proceeding with master startup.
I20250903 08:08:11.453487 16813 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f8994423f61427b84bb24698530b26b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f8994423f61427b84bb24698530b26b. Latest consensus state: current_term: 1 leader_uuid: "8f8994423f61427b84bb24698530b26b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f8994423f61427b84bb24698530b26b" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 43079 } } }
I20250903 08:08:11.453506 16812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f8994423f61427b84bb24698530b26b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f8994423f61427b84bb24698530b26b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f8994423f61427b84bb24698530b26b" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 43079 } } }
I20250903 08:08:11.453600 16813 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f8994423f61427b84bb24698530b26b [sys.catalog]: This master's current role is: LEADER
I20250903 08:08:11.453609 16812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f8994423f61427b84bb24698530b26b [sys.catalog]: This master's current role is: LEADER
I20250903 08:08:11.454316 16817 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250903 08:08:11.457269 16817 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250903 08:08:11.462620 16817 catalog_manager.cc:1349] Generated new cluster ID: 210c7435f94241c4b1b4ce5bbbd8109d
I20250903 08:08:11.462678 16817 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250903 08:08:11.469298 16817 catalog_manager.cc:1372] Generated new certificate authority record
I20250903 08:08:11.470232 16817 catalog_manager.cc:1506] Loading token signing keys...
I20250903 08:08:11.474274 16817 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 8f8994423f61427b84bb24698530b26b: Generated new TSK 0
I20250903 08:08:11.475046 16817 catalog_manager.cc:1516] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.201.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.201.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.201.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.201.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.201.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.201.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.201.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.201.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/krb5kdc/kudu.keytab.
I20250903 08:08:11.523604 13095 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUML90r/build/debug/bin/kudu
/tmp/dist-test-taskUML90r/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.201.193:0
--local_ip_for_outbound_sockets=127.12.201.193
--webserver_interface=127.12.201.193
--webserver_port=0
--tserver_master_addrs=127.12.201.254:43079
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.201.212:42945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 08:08:11.624603 16834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:08:11.624815 16834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:08:11.624869 16834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:08:11.627971 16834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 08:08:11.628028 16834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:08:11.628135 16834 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.201.193
I20250903 08:08:11.631914 16834 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.201.212:42945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.201.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.201.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.12.201.254:43079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16834
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.201.193
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 07:50:08 UTC on 5fd53c4cbb9d
build id 7874
I20250903 08:08:11.632858 16834 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:08:11.633642 16834 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:08:11.639645 16839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:11.639653 16840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:11.639652 16842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:08:11.639873 16834 server_base.cc:1047] running on GCE node
I20250903 08:08:11.640266 16834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:08:11.640741 16834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:08:11.641880 16834 hybrid_clock.cc:648] HybridClock initialized: now 1756886891641869 us; error 23 us; skew 500 ppm
Sep 03 08:08:11 dist-test-slave-27r9 krb5kdc[16722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886891, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 08:08:11.644439 16834 init.cc:377] Logged in from keytab as kudu/127.12.201.193@KRBTEST.COM (short username kudu)
I20250903 08:08:11.645375 16834 webserver.cc:480] Webserver started at http://127.12.201.193:38351/ using document root <none> and password file <none>
I20250903 08:08:11.645879 16834 fs_manager.cc:362] Metadata directory not provided
I20250903 08:08:11.645928 16834 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:08:11.646106 16834 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:08:11.647794 16834 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-0/data/instance:
uuid: "a1918b67d8854562a5fb036d0ad4882c"
format_stamp: "Formatted at 2025-09-03 08:08:11 on dist-test-slave-27r9"
server_key: "48a3f1b4a4b6516d795235bef182f7b4"
server_key_iv: "5118e247d1dac38cba15afee151d3ff4"
server_key_version: "encryptionkey@0"
I20250903 08:08:11.648237 16834 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-0/wal/instance:
uuid: "a1918b67d8854562a5fb036d0ad4882c"
format_stamp: "Formatted at 2025-09-03 08:08:11 on dist-test-slave-27r9"
server_key: "48a3f1b4a4b6516d795235bef182f7b4"
server_key_iv: "5118e247d1dac38cba15afee151d3ff4"
server_key_version: "encryptionkey@0"
I20250903 08:08:11.651360 16834 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250903 08:08:11.653506 16849 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:11.654413 16834 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250903 08:08:11.654505 16834 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-0/wal
uuid: "a1918b67d8854562a5fb036d0ad4882c"
format_stamp: "Formatted at 2025-09-03 08:08:11 on dist-test-slave-27r9"
server_key: "48a3f1b4a4b6516d795235bef182f7b4"
server_key_iv: "5118e247d1dac38cba15afee151d3ff4"
server_key_version: "encryptionkey@0"
I20250903 08:08:11.654599 16834 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:08:11.672796 16834 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:08:11.675770 16834 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:08:11.675951 16834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:08:11.676481 16834 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 08:08:11.677291 16834 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 08:08:11.677337 16834 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:11.677381 16834 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 08:08:11.677414 16834 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:11.685807 16834 rpc_server.cc:307] RPC server started. Bound to: 127.12.201.193:45249
I20250903 08:08:11.685843 16962 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.201.193:45249 every 8 connection(s)
I20250903 08:08:11.686704 16834 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-0/data/info.pb
I20250903 08:08:11.688510 13095 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUML90r/build/debug/bin/kudu as pid 16834
I20250903 08:08:11.688624 13095 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-0/wal/instance
I20250903 08:08:11.688860 13095 external_mini_cluster.cc:1468] Setting key 6289db9e8e9c7b4753781f94dba8dd9e
Sep 03 08:08:11 dist-test-slave-27r9 krb5kdc[16722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886891, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.193@KRBTEST.COM for kudu/127.12.201.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.201.194@KRBTEST.COM; defaulting to no policy
I20250903 08:08:11.699580 16809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:11.688364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.193:44285 (local address 127.12.201.254:43079)
0903 08:08:11.688542 (+ 178us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:11.688547 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:11.689209 (+ 662us) server_negotiation.cc:408] Connection header received
0903 08:08:11.690144 (+ 935us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:11.690148 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:11.690209 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:11.690287 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:11.692356 (+ 2069us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:11.693137 (+ 781us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:11.693727 (+ 590us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:11.693973 (+ 246us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:11.696301 (+ 2328us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:11.696316 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:11.696318 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:08:11.696340 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:11.697899 (+ 1559us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:11.698456 (+ 557us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:11.698459 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:11.698460 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:11.698497 (+ 37us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:11.698699 (+ 202us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:11.698701 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:11.698703 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:11.698846 (+ 143us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:11.698912 (+ 66us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:11.699370 (+ 458us) server_negotiation.cc:300] Negotiation successful
0903 08:08:11.699464 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20250903 08:08:11.700176 16965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:11.688570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:43079 (local address 127.12.201.193:44285)
0903 08:08:11.689082 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:11.689124 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:11.689966 (+ 842us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:11.690444 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:11.690454 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:11.691035 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:08:11.692216 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:11.692231 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:11.693253 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:11.693255 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:11.693622 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:11.693629 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:11.693796 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:11.694329 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:08:11.694351 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:08:11.696159 (+ 1808us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:08:11.698022 (+ 1863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:11.698030 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:11.698043 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:11.698370 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:11.698562 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:11.698564 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:11.698566 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:11.698638 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:11.698915 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:08:11.698920 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:08:11.699167 (+ 247us) client_negotiation.cc:769] Sending connection context
0903 08:08:11.699324 (+ 157us) client_negotiation.cc:240] Negotiation successful
0903 08:08:11.699514 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":124,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.201.194@KRBTEST.COM" created.
I20250903 08:08:11.701252 16963 heartbeater.cc:344] Connected to a master server at 127.12.201.254:43079
I20250903 08:08:11.701462 16963 heartbeater.cc:461] Registering TS with master...
I20250903 08:08:11.701898 16963 heartbeater.cc:507] Master 127.12.201.254:43079 requested a full tablet report, sending...
I20250903 08:08:11.703380 16770 ts_manager.cc:194] Registered new tserver with Master: a1918b67d8854562a5fb036d0ad4882c (127.12.201.193:45249)
I20250903 08:08:11.705075 16770 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.201.193@KRBTEST.COM'} at 127.12.201.193:44285
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.201.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.201.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.201.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.201.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.201.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.201.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/krb5kdc/kudu.keytab.
I20250903 08:08:11.735452 13095 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUML90r/build/debug/bin/kudu
/tmp/dist-test-taskUML90r/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.201.194:0
--local_ip_for_outbound_sockets=127.12.201.194
--webserver_interface=127.12.201.194
--webserver_port=0
--tserver_master_addrs=127.12.201.254:43079
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.201.212:42945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 08:08:11.835225 16970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:08:11.835419 16970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:08:11.835510 16970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:08:11.838557 16970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 08:08:11.838609 16970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:08:11.838721 16970 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.201.194
I20250903 08:08:11.842598 16970 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.201.212:42945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.201.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.201.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.12.201.254:43079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.201.194
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 07:50:08 UTC on 5fd53c4cbb9d
build id 7874
I20250903 08:08:11.843549 16970 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:08:11.844360 16970 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:08:11.850287 16975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:11.850284 16976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:08:11.850463 16970 server_base.cc:1047] running on GCE node
W20250903 08:08:11.850311 16978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:08:11.850939 16970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:08:11.851486 16970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:08:11.852656 16970 hybrid_clock.cc:648] HybridClock initialized: now 1756886891852644 us; error 29 us; skew 500 ppm
Sep 03 08:08:11 dist-test-slave-27r9 krb5kdc[16722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886891, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 08:08:11.855211 16970 init.cc:377] Logged in from keytab as kudu/127.12.201.194@KRBTEST.COM (short username kudu)
I20250903 08:08:11.856141 16970 webserver.cc:480] Webserver started at http://127.12.201.194:43051/ using document root <none> and password file <none>
I20250903 08:08:11.856626 16970 fs_manager.cc:362] Metadata directory not provided
I20250903 08:08:11.856668 16970 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:08:11.856822 16970 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:08:11.858505 16970 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-1/data/instance:
uuid: "00fa0a24383e4510afb3394690921a49"
format_stamp: "Formatted at 2025-09-03 08:08:11 on dist-test-slave-27r9"
server_key: "846a1a9732d13aa55b461eba132ae931"
server_key_iv: "7660e5ce9f71a082f681e6cc7845eab0"
server_key_version: "encryptionkey@0"
I20250903 08:08:11.858933 16970 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-1/wal/instance:
uuid: "00fa0a24383e4510afb3394690921a49"
format_stamp: "Formatted at 2025-09-03 08:08:11 on dist-test-slave-27r9"
server_key: "846a1a9732d13aa55b461eba132ae931"
server_key_iv: "7660e5ce9f71a082f681e6cc7845eab0"
server_key_version: "encryptionkey@0"
I20250903 08:08:11.862239 16970 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20250903 08:08:11.864322 16985 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:11.865236 16970 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250903 08:08:11.865321 16970 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-1/wal
uuid: "00fa0a24383e4510afb3394690921a49"
format_stamp: "Formatted at 2025-09-03 08:08:11 on dist-test-slave-27r9"
server_key: "846a1a9732d13aa55b461eba132ae931"
server_key_iv: "7660e5ce9f71a082f681e6cc7845eab0"
server_key_version: "encryptionkey@0"
I20250903 08:08:11.865391 16970 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:08:11.879264 16970 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:08:11.881927 16970 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:08:11.882098 16970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:08:11.882529 16970 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 08:08:11.883314 16970 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 08:08:11.883361 16970 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:11.883399 16970 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 08:08:11.883420 16970 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:11.891584 16970 rpc_server.cc:307] RPC server started. Bound to: 127.12.201.194:43911
I20250903 08:08:11.891609 17098 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.201.194:43911 every 8 connection(s)
I20250903 08:08:11.892467 16970 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-1/data/info.pb
Sep 03 08:08:11 dist-test-slave-27r9 krb5kdc[16722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886891, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.194@KRBTEST.COM for kudu/127.12.201.254@KRBTEST.COM
I20250903 08:08:11.900352 13095 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUML90r/build/debug/bin/kudu as pid 16970
I20250903 08:08:11.900454 13095 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-1/wal/instance
I20250903 08:08:11.900686 13095 external_mini_cluster.cc:1468] Setting key ae4030bd18fb108f716c34903900c31b
I20250903 08:08:11.904072 16809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:11.894006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.194:36415 (local address 127.12.201.254:43079)
0903 08:08:11.894153 (+ 147us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:11.894157 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:11.894736 (+ 579us) server_negotiation.cc:408] Connection header received
0903 08:08:11.895583 (+ 847us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:11.895587 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:11.895648 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:11.895723 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:11.897280 (+ 1557us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:11.897771 (+ 491us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:11.898437 (+ 666us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:11.898613 (+ 176us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:11.900776 (+ 2163us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:11.900796 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:11.900799 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:08:11.900827 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:11.902336 (+ 1509us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:11.902755 (+ 419us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:11.902758 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:11.902759 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:11.902804 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:11.903076 (+ 272us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:11.903081 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:11.903083 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:11.903254 (+ 171us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:11.903346 (+ 92us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:11.903807 (+ 461us) server_negotiation.cc:300] Negotiation successful
0903 08:08:11.903924 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20250903 08:08:11.904592 17101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:11.894222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:43079 (local address 127.12.201.194:36415)
0903 08:08:11.894600 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:11.894635 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:11.895397 (+ 762us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:11.895879 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:11.895887 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:11.896239 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:08:11.897134 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:11.897148 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:11.897884 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:11.897887 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:11.898324 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:11.898332 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:11.898512 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:11.899127 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:08:11.899148 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:08:11.900632 (+ 1484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:08:11.902446 (+ 1814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:11.902451 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:11.902461 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:11.902666 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:11.902880 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:11.902885 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:11.902887 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:11.902958 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:11.903343 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:08:11.903348 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:08:11.903562 (+ 214us) client_negotiation.cc:769] Sending connection context
0903 08:08:11.903730 (+ 168us) client_negotiation.cc:240] Negotiation successful
0903 08:08:11.903960 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":100,"threads_started":1}
I20250903 08:08:11.905615 17099 heartbeater.cc:344] Connected to a master server at 127.12.201.254:43079
I20250903 08:08:11.905834 17099 heartbeater.cc:461] Registering TS with master...
I20250903 08:08:11.906262 17099 heartbeater.cc:507] Master 127.12.201.254:43079 requested a full tablet report, sending...
I20250903 08:08:11.907135 16770 ts_manager.cc:194] Registered new tserver with Master: 00fa0a24383e4510afb3394690921a49 (127.12.201.194:43911)
I20250903 08:08:11.907648 16770 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.201.194@KRBTEST.COM'} at 127.12.201.194:36415
WARNING: no policy specified for kudu/127.12.201.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.201.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.201.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.201.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.201.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.201.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.201.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.201.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/krb5kdc/kudu.keytab.
I20250903 08:08:11.943638 13095 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUML90r/build/debug/bin/kudu
/tmp/dist-test-taskUML90r/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.201.195:0
--local_ip_for_outbound_sockets=127.12.201.195
--webserver_interface=127.12.201.195
--webserver_port=0
--tserver_master_addrs=127.12.201.254:43079
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.201.212:42945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 08:08:12.044024 17106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:08:12.044232 17106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:08:12.044294 17106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:08:12.047503 17106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 08:08:12.047561 17106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:08:12.047693 17106 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.201.195
I20250903 08:08:12.051810 17106 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.201.212:42945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.201.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.201.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.12.201.254:43079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17106
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.201.195
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 07:50:08 UTC on 5fd53c4cbb9d
build id 7874
I20250903 08:08:12.052803 17106 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:08:12.053635 17106 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:08:12.060441 17112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:12.060426 17111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:12.060683 17114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:08:12.060766 17106 server_base.cc:1047] running on GCE node
I20250903 08:08:12.061393 17106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:08:12.062033 17106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:08:12.063181 17106 hybrid_clock.cc:648] HybridClock initialized: now 1756886892063168 us; error 31 us; skew 500 ppm
Sep 03 08:08:12 dist-test-slave-27r9 krb5kdc[16722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886892, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 08:08:12.066143 17106 init.cc:377] Logged in from keytab as kudu/127.12.201.195@KRBTEST.COM (short username kudu)
I20250903 08:08:12.067138 17106 webserver.cc:480] Webserver started at http://127.12.201.195:37953/ using document root <none> and password file <none>
I20250903 08:08:12.067685 17106 fs_manager.cc:362] Metadata directory not provided
I20250903 08:08:12.067734 17106 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:08:12.067914 17106 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:08:12.069646 17106 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-2/data/instance:
uuid: "9776ec513cf242b291a8fc9853266f0a"
format_stamp: "Formatted at 2025-09-03 08:08:12 on dist-test-slave-27r9"
server_key: "e0cc1a912b57f89d15a6a6489582d863"
server_key_iv: "af80e69314daa757c0a26e010a383035"
server_key_version: "encryptionkey@0"
I20250903 08:08:12.070129 17106 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-2/wal/instance:
uuid: "9776ec513cf242b291a8fc9853266f0a"
format_stamp: "Formatted at 2025-09-03 08:08:12 on dist-test-slave-27r9"
server_key: "e0cc1a912b57f89d15a6a6489582d863"
server_key_iv: "af80e69314daa757c0a26e010a383035"
server_key_version: "encryptionkey@0"
I20250903 08:08:12.073346 17106 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20250903 08:08:12.075435 17121 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:12.076361 17106 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250903 08:08:12.076460 17106 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-2/wal
uuid: "9776ec513cf242b291a8fc9853266f0a"
format_stamp: "Formatted at 2025-09-03 08:08:12 on dist-test-slave-27r9"
server_key: "e0cc1a912b57f89d15a6a6489582d863"
server_key_iv: "af80e69314daa757c0a26e010a383035"
server_key_version: "encryptionkey@0"
I20250903 08:08:12.076545 17106 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:08:12.089483 17106 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:08:12.092095 17106 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:08:12.092262 17106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:08:12.092765 17106 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 08:08:12.093564 17106 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 08:08:12.093607 17106 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:12.093660 17106 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 08:08:12.093684 17106 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:12.102123 17106 rpc_server.cc:307] RPC server started. Bound to: 127.12.201.195:45865
I20250903 08:08:12.102170 17234 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.201.195:45865 every 8 connection(s)
I20250903 08:08:12.103001 17106 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-2/data/info.pb
I20250903 08:08:12.108901 13095 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUML90r/build/debug/bin/kudu as pid 17106
I20250903 08:08:12.109026 13095 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756886860253814-13095-0/minicluster-data/ts-2/wal/instance
I20250903 08:08:12.109289 13095 external_mini_cluster.cc:1468] Setting key cae630bb017dd2b73f8c8c62bfa8f249
Sep 03 08:08:12 dist-test-slave-27r9 krb5kdc[16722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886892, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.195@KRBTEST.COM for kudu/127.12.201.254@KRBTEST.COM
I20250903 08:08:12.114367 16809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:12.104657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.195:56801 (local address 127.12.201.254:43079)
0903 08:08:12.104791 (+ 134us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:12.104795 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:12.105419 (+ 624us) server_negotiation.cc:408] Connection header received
0903 08:08:12.106150 (+ 731us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:12.106153 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:12.106203 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:12.106275 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:12.107830 (+ 1555us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:12.108314 (+ 484us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:12.108924 (+ 610us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:12.109129 (+ 205us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:12.111365 (+ 2236us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:12.111379 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:12.111381 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:08:12.111405 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:12.112643 (+ 1238us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:12.113100 (+ 457us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:12.113104 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:12.113105 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:12.113146 (+ 41us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:12.113397 (+ 251us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:12.113400 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:12.113402 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:12.113543 (+ 141us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:12.113632 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:12.114085 (+ 453us) server_negotiation.cc:300] Negotiation successful
0903 08:08:12.114204 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20250903 08:08:12.114931 17237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:12.104889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:43079 (local address 127.12.201.195:56801)
0903 08:08:12.105278 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:12.105311 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:12.105975 (+ 664us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:12.106402 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:12.106410 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:12.106754 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:08:12.107698 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:12.107712 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:12.108403 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:12.108405 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:12.108809 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:12.108815 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:12.108952 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:12.109570 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:08:12.109605 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:08:12.111221 (+ 1616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:08:12.112755 (+ 1534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:12.112761 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:12.112772 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:12.113004 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:12.113226 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:12.113231 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:12.113234 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:12.113313 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:12.113640 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:08:12.113649 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:08:12.113859 (+ 210us) client_negotiation.cc:769] Sending connection context
0903 08:08:12.114049 (+ 190us) client_negotiation.cc:240] Negotiation successful
0903 08:08:12.114301 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":112,"threads_started":1}
I20250903 08:08:12.115888 17235 heartbeater.cc:344] Connected to a master server at 127.12.201.254:43079
I20250903 08:08:12.116120 17235 heartbeater.cc:461] Registering TS with master...
I20250903 08:08:12.116591 17235 heartbeater.cc:507] Master 127.12.201.254:43079 requested a full tablet report, sending...
I20250903 08:08:12.117456 16770 ts_manager.cc:194] Registered new tserver with Master: 9776ec513cf242b291a8fc9853266f0a (127.12.201.195:45865)
I20250903 08:08:12.117987 16770 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.201.195@KRBTEST.COM'} at 127.12.201.195:56801
I20250903 08:08:12.122735 13095 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Sep 03 08:08:12 dist-test-slave-27r9 krb5kdc[16722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886892, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 03 08:08:12 dist-test-slave-27r9 krb5kdc[16722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886892, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.12.201.254@KRBTEST.COM
I20250903 08:08:12.145277 16809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:12.137548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33558 (local address 127.12.201.254:43079)
0903 08:08:12.137703 (+ 155us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:12.137707 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:12.137807 (+ 100us) server_negotiation.cc:408] Connection header received
0903 08:08:12.137947 (+ 140us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:12.137951 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:12.138000 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:12.138067 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:12.138874 (+ 807us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:12.139435 (+ 561us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:12.140050 (+ 615us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:12.140256 (+ 206us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:12.142395 (+ 2139us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:12.142412 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:12.142414 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:08:12.142440 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:12.143790 (+ 1350us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:12.144192 (+ 402us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:12.144195 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:12.144196 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:12.144239 (+ 43us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:12.144449 (+ 210us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:12.144453 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:12.144455 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:12.144623 (+ 168us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:12.144750 (+ 127us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:12.145005 (+ 255us) server_negotiation.cc:300] Negotiation successful
0903 08:08:12.145121 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20250903 08:08:12.145625 16770 server_base.cc:1129] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:33558
I20250903 08:08:12.146353 13095 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUML90r/build/debug/bin/kudu with pid 16834
I20250903 08:08:12.152318 13095 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUML90r/build/debug/bin/kudu with pid 16970
I20250903 08:08:12.157883 13095 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUML90r/build/debug/bin/kudu with pid 17106
I20250903 08:08:12.162973 13095 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUML90r/build/debug/bin/kudu with pid 16738
2025-09-03T08:08:12Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2990 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 03 08:08:12 dist-test-slave-27r9 krb5kdc[17247](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 03 08:08:12 dist-test-slave-27r9 krb5kdc[17247](info): set up 2 sockets
Sep 03 08:08:12 dist-test-slave-27r9 krb5kdc[17247](info): commencing operation
krb5kdc: starting...
W20250903 08:08:14.139523 13095 mini_kdc.cc:121] Time spent starting KDC: real 1.955s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/krb5kdc/test-user.keytab.
Sep 03 08:08:14 dist-test-slave-27r9 krb5kdc[17247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886894, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-09-03T08:08:14Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-03T08:08:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.201.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.201.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.201.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.201.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.201.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.201.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.201.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.201.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/krb5kdc/kudu.keytab.
I20250903 08:08:14.274436 13095 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUML90r/build/debug/bin/kudu
/tmp/dist-test-taskUML90r/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.201.254:42407
--webserver_interface=127.12.201.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.201.212:41681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.201.254:42407
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 08:08:14.380810 17263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:08:14.381067 17263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:08:14.381156 17263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:08:14.384418 17263 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250903 08:08:14.384478 17263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 08:08:14.384518 17263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:08:14.384552 17263 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250903 08:08:14.384585 17263 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250903 08:08:14.388844 17263 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.201.212:41681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.201.254:42407
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.201.254:42407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.201.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17263
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 07:50:08 UTC on 5fd53c4cbb9d
build id 7874
I20250903 08:08:14.389925 17263 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:08:14.390914 17263 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:08:14.396346 17271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:14.396366 17268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:14.396453 17269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:08:14.396409 17263 server_base.cc:1047] running on GCE node
I20250903 08:08:14.397043 17263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:08:14.397888 17263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:08:14.399039 17263 hybrid_clock.cc:648] HybridClock initialized: now 1756886894399029 us; error 30 us; skew 500 ppm
Sep 03 08:08:14 dist-test-slave-27r9 krb5kdc[17247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886894, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 08:08:14.401500 17263 init.cc:377] Logged in from keytab as kudu/127.12.201.254@KRBTEST.COM (short username kudu)
I20250903 08:08:14.402453 17263 webserver.cc:480] Webserver started at http://127.12.201.254:41957/ using document root <none> and password file <none>
I20250903 08:08:14.403148 17263 fs_manager.cc:362] Metadata directory not provided
I20250903 08:08:14.403210 17263 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:08:14.403432 17263 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:08:14.405144 17263 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/minicluster-data/master-0/data/instance:
uuid: "9791408f513f4ef8a2bc00cc0be9b0dd"
format_stamp: "Formatted at 2025-09-03 08:08:14 on dist-test-slave-27r9"
server_key: "6b4e18f41ac296e662092e1b8c537366"
server_key_iv: "ae59becff74778b4bcd017915a15a512"
server_key_version: "encryptionkey@0"
I20250903 08:08:14.405588 17263 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/minicluster-data/master-0/wal/instance:
uuid: "9791408f513f4ef8a2bc00cc0be9b0dd"
format_stamp: "Formatted at 2025-09-03 08:08:14 on dist-test-slave-27r9"
server_key: "6b4e18f41ac296e662092e1b8c537366"
server_key_iv: "ae59becff74778b4bcd017915a15a512"
server_key_version: "encryptionkey@0"
I20250903 08:08:14.408782 17263 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20250903 08:08:14.410778 17278 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:14.411751 17263 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250903 08:08:14.411854 17263 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/minicluster-data/master-0/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/minicluster-data/master-0/wal
uuid: "9791408f513f4ef8a2bc00cc0be9b0dd"
format_stamp: "Formatted at 2025-09-03 08:08:14 on dist-test-slave-27r9"
server_key: "6b4e18f41ac296e662092e1b8c537366"
server_key_iv: "ae59becff74778b4bcd017915a15a512"
server_key_version: "encryptionkey@0"
I20250903 08:08:14.411940 17263 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:08:14.422751 17263 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:08:14.425309 17263 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:08:14.425477 17263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:08:14.432257 17330 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.201.254:42407 every 8 connection(s)
I20250903 08:08:14.432258 17263 rpc_server.cc:307] RPC server started. Bound to: 127.12.201.254:42407
I20250903 08:08:14.433228 17263 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/minicluster-data/master-0/data/info.pb
I20250903 08:08:14.435752 17331 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:08:14.440709 13095 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUML90r/build/debug/bin/kudu as pid 17263
I20250903 08:08:14.440833 13095 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/minicluster-data/master-0/wal/instance
I20250903 08:08:14.440958 17331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9791408f513f4ef8a2bc00cc0be9b0dd: Bootstrap starting.
I20250903 08:08:14.441074 13095 external_mini_cluster.cc:1468] Setting key 416432de30e8bccc48230431a679594c
I20250903 08:08:14.443339 17331 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9791408f513f4ef8a2bc00cc0be9b0dd: Neither blocks nor log segments found. Creating new log.
I20250903 08:08:14.444031 17331 log.cc:826] T 00000000000000000000000000000000 P 9791408f513f4ef8a2bc00cc0be9b0dd: Log is configured to *not* fsync() on all Append() calls
I20250903 08:08:14.445680 17331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9791408f513f4ef8a2bc00cc0be9b0dd: No bootstrap required, opened a new log
Sep 03 08:08:14 dist-test-slave-27r9 krb5kdc[17247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886894, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.201.254@KRBTEST.COM
I20250903 08:08:14.448236 17331 raft_consensus.cc:357] T 00000000000000000000000000000000 P 9791408f513f4ef8a2bc00cc0be9b0dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9791408f513f4ef8a2bc00cc0be9b0dd" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 42407 } }
I20250903 08:08:14.448418 17331 raft_consensus.cc:383] T 00000000000000000000000000000000 P 9791408f513f4ef8a2bc00cc0be9b0dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:08:14.448479 17331 raft_consensus.cc:738] T 00000000000000000000000000000000 P 9791408f513f4ef8a2bc00cc0be9b0dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9791408f513f4ef8a2bc00cc0be9b0dd, State: Initialized, Role: FOLLOWER
I20250903 08:08:14.448974 17331 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9791408f513f4ef8a2bc00cc0be9b0dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9791408f513f4ef8a2bc00cc0be9b0dd" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 42407 } }
I20250903 08:08:14.449119 17331 raft_consensus.cc:397] T 00000000000000000000000000000000 P 9791408f513f4ef8a2bc00cc0be9b0dd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250903 08:08:14.449180 17331 raft_consensus.cc:491] T 00000000000000000000000000000000 P 9791408f513f4ef8a2bc00cc0be9b0dd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250903 08:08:14.449321 17331 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 9791408f513f4ef8a2bc00cc0be9b0dd [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:08:14.450134 17331 raft_consensus.cc:513] T 00000000000000000000000000000000 P 9791408f513f4ef8a2bc00cc0be9b0dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9791408f513f4ef8a2bc00cc0be9b0dd" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 42407 } }
I20250903 08:08:14.450443 17331 leader_election.cc:304] T 00000000000000000000000000000000 P 9791408f513f4ef8a2bc00cc0be9b0dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9791408f513f4ef8a2bc00cc0be9b0dd; no voters:
I20250903 08:08:14.450755 17331 leader_election.cc:290] T 00000000000000000000000000000000 P 9791408f513f4ef8a2bc00cc0be9b0dd [CANDIDATE]: Term 1 election: Requested vote from peers
I20250903 08:08:14.451081 17336 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 9791408f513f4ef8a2bc00cc0be9b0dd [term 1 FOLLOWER]: Leader election won for term 1
I20250903 08:08:14.451295 17336 raft_consensus.cc:695] T 00000000000000000000000000000000 P 9791408f513f4ef8a2bc00cc0be9b0dd [term 1 LEADER]: Becoming Leader. State: Replica: 9791408f513f4ef8a2bc00cc0be9b0dd, State: Running, Role: LEADER
I20250903 08:08:14.451556 17336 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9791408f513f4ef8a2bc00cc0be9b0dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9791408f513f4ef8a2bc00cc0be9b0dd" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 42407 } }
I20250903 08:08:14.451920 17331 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9791408f513f4ef8a2bc00cc0be9b0dd [sys.catalog]: configured and running, proceeding with master startup.
I20250903 08:08:14.452602 17336 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9791408f513f4ef8a2bc00cc0be9b0dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9791408f513f4ef8a2bc00cc0be9b0dd. Latest consensus state: current_term: 1 leader_uuid: "9791408f513f4ef8a2bc00cc0be9b0dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9791408f513f4ef8a2bc00cc0be9b0dd" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 42407 } } }
I20250903 08:08:14.452926 17337 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9791408f513f4ef8a2bc00cc0be9b0dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9791408f513f4ef8a2bc00cc0be9b0dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9791408f513f4ef8a2bc00cc0be9b0dd" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 42407 } } }
I20250903 08:08:14.453141 17337 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9791408f513f4ef8a2bc00cc0be9b0dd [sys.catalog]: This master's current role is: LEADER
I20250903 08:08:14.453401 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:14.442208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46290 (local address 127.12.201.254:42407)
0903 08:08:14.442606 (+ 398us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:14.442615 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:14.442647 (+ 32us) server_negotiation.cc:408] Connection header received
0903 08:08:14.443231 (+ 584us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:14.443250 (+ 19us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:14.443569 (+ 319us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:14.443868 (+ 299us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:14.444758 (+ 890us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:14.445490 (+ 732us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:14.446158 (+ 668us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:14.446452 (+ 294us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:14.448624 (+ 2172us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:14.448645 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:14.448658 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:08:14.448709 (+ 51us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:14.450609 (+ 1900us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:14.451120 (+ 511us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:14.451126 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:14.451131 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:14.451208 (+ 77us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:14.451453 (+ 245us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:14.451456 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:14.451458 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:14.451884 (+ 426us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:14.452086 (+ 202us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:14.452316 (+ 230us) server_negotiation.cc:300] Negotiation successful
0903 08:08:14.452569 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":139,"threads_started":1}
I20250903 08:08:14.453526 17345 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250903 08:08:14.452819 17336 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9791408f513f4ef8a2bc00cc0be9b0dd [sys.catalog]: This master's current role is: LEADER
I20250903 08:08:14.456490 17345 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250903 08:08:14.461278 17345 catalog_manager.cc:1349] Generated new cluster ID: 1124d2cdb81141c68cf0dd7c0c420346
I20250903 08:08:14.461345 17345 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250903 08:08:14.467685 17345 catalog_manager.cc:1372] Generated new certificate authority record
I20250903 08:08:14.468784 17345 catalog_manager.cc:1506] Loading token signing keys...
I20250903 08:08:14.478225 17345 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 9791408f513f4ef8a2bc00cc0be9b0dd: Generated new TSK 0
I20250903 08:08:14.479110 17345 catalog_manager.cc:1516] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.201.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.201.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.201.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.201.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.201.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.201.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.201.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.201.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/krb5kdc/kudu.keytab.
I20250903 08:08:14.529561 13095 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUML90r/build/debug/bin/kudu
/tmp/dist-test-taskUML90r/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.201.193:0
--local_ip_for_outbound_sockets=127.12.201.193
--webserver_interface=127.12.201.193
--webserver_port=0
--tserver_master_addrs=127.12.201.254:42407
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.201.212:41681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 08:08:14.633741 17359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:08:14.633991 17359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:08:14.634075 17359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:08:14.637223 17359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 08:08:14.637280 17359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:08:14.637356 17359 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.201.193
I20250903 08:08:14.641356 17359 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.201.212:41681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.201.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.201.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.12.201.254:42407
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17359
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.201.193
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 07:50:08 UTC on 5fd53c4cbb9d
build id 7874
I20250903 08:08:14.642288 17359 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:08:14.643079 17359 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:08:14.649125 17365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:14.649170 17367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:14.649169 17364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:08:14.649441 17359 server_base.cc:1047] running on GCE node
I20250903 08:08:14.649787 17359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:08:14.650249 17359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:08:14.651392 17359 hybrid_clock.cc:648] HybridClock initialized: now 1756886894651378 us; error 29 us; skew 500 ppm
Sep 03 08:08:14 dist-test-slave-27r9 krb5kdc[17247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886894, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 08:08:14.654235 17359 init.cc:377] Logged in from keytab as kudu/127.12.201.193@KRBTEST.COM (short username kudu)
I20250903 08:08:14.655263 17359 webserver.cc:480] Webserver started at http://127.12.201.193:35625/ using document root <none> and password file <none>
I20250903 08:08:14.655761 17359 fs_manager.cc:362] Metadata directory not provided
I20250903 08:08:14.655803 17359 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:08:14.655951 17359 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:08:14.657618 17359 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/minicluster-data/ts-0/data/instance:
uuid: "52295618a7544f119d410caddd02a16d"
format_stamp: "Formatted at 2025-09-03 08:08:14 on dist-test-slave-27r9"
server_key: "6f2900194fe9f3822c74ea3c1bd6afe2"
server_key_iv: "177ee59a783b54a087c6dd3a96590ab5"
server_key_version: "encryptionkey@0"
I20250903 08:08:14.658043 17359 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/minicluster-data/ts-0/wal/instance:
uuid: "52295618a7544f119d410caddd02a16d"
format_stamp: "Formatted at 2025-09-03 08:08:14 on dist-test-slave-27r9"
server_key: "6f2900194fe9f3822c74ea3c1bd6afe2"
server_key_iv: "177ee59a783b54a087c6dd3a96590ab5"
server_key_version: "encryptionkey@0"
I20250903 08:08:14.661242 17359 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20250903 08:08:14.663255 17374 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:14.664216 17359 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250903 08:08:14.664314 17359 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/minicluster-data/ts-0/wal
uuid: "52295618a7544f119d410caddd02a16d"
format_stamp: "Formatted at 2025-09-03 08:08:14 on dist-test-slave-27r9"
server_key: "6f2900194fe9f3822c74ea3c1bd6afe2"
server_key_iv: "177ee59a783b54a087c6dd3a96590ab5"
server_key_version: "encryptionkey@0"
I20250903 08:08:14.664403 17359 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:08:14.703424 17359 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:08:14.706398 17359 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:08:14.706574 17359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:08:14.707113 17359 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 08:08:14.707940 17359 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 08:08:14.707986 17359 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:14.708042 17359 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 08:08:14.708074 17359 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:14.716652 17359 rpc_server.cc:307] RPC server started. Bound to: 127.12.201.193:43541
I20250903 08:08:14.716691 17487 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.201.193:43541 every 8 connection(s)
I20250903 08:08:14.717521 17359 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/minicluster-data/ts-0/data/info.pb
Sep 03 08:08:14 dist-test-slave-27r9 krb5kdc[17247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886894, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.193@KRBTEST.COM for kudu/127.12.201.254@KRBTEST.COM
I20250903 08:08:14.725515 13095 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUML90r/build/debug/bin/kudu as pid 17359
I20250903 08:08:14.725634 13095 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/minicluster-data/ts-0/wal/instance
I20250903 08:08:14.725896 13095 external_mini_cluster.cc:1468] Setting key 45032a3365c3d9a8065ec01631fc85c8
I20250903 08:08:14.729974 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:14.719193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.193:41401 (local address 127.12.201.254:42407)
0903 08:08:14.719321 (+ 128us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:14.719325 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:14.719935 (+ 610us) server_negotiation.cc:408] Connection header received
0903 08:08:14.720783 (+ 848us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:14.720787 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:14.720845 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:14.720924 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:14.722124 (+ 1200us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:14.722623 (+ 499us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:14.723733 (+ 1110us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:14.723906 (+ 173us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:14.726123 (+ 2217us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:14.726140 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:14.726143 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:08:14.726166 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:14.727560 (+ 1394us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:14.728214 (+ 654us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:14.728219 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:14.728224 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:14.728287 (+ 63us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:14.728634 (+ 347us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:14.728640 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:14.728643 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:14.728865 (+ 222us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:14.728972 (+ 107us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:14.729586 (+ 614us) server_negotiation.cc:300] Negotiation successful
0903 08:08:14.729732 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20250903 08:08:14.730499 17490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:14.719418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:42407 (local address 127.12.201.193:41401)
0903 08:08:14.719797 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:14.719830 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:14.720572 (+ 742us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:14.721087 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:14.721095 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:14.721451 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:08:14.721976 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:14.721987 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:14.722745 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:14.722748 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:14.723612 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:14.723621 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:14.723804 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:14.724474 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:08:14.724498 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:08:14.725983 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:08:14.727718 (+ 1735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:14.727726 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:14.727739 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:14.728094 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:14.728403 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:14.728409 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:14.728414 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:14.728531 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:14.728988 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:08:14.728997 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:08:14.729283 (+ 286us) client_negotiation.cc:769] Sending connection context
0903 08:08:14.729523 (+ 240us) client_negotiation.cc:240] Negotiation successful
0903 08:08:14.729768 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":95,"threads_started":1}
I20250903 08:08:14.731961 17488 heartbeater.cc:344] Connected to a master server at 127.12.201.254:42407
I20250903 08:08:14.732242 17488 heartbeater.cc:461] Registering TS with master...
I20250903 08:08:14.732900 17488 heartbeater.cc:507] Master 127.12.201.254:42407 requested a full tablet report, sending...
I20250903 08:08:14.734544 17295 ts_manager.cc:194] Registered new tserver with Master: 52295618a7544f119d410caddd02a16d (127.12.201.193:43541)
WARNING: no policy specified for kudu/127.12.201.194@KRBTEST.COM; defaulting to no policy
I20250903 08:08:14.736212 17295 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.201.193@KRBTEST.COM'} at 127.12.201.193:41401
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.201.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.201.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.201.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.201.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.201.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.201.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.201.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/krb5kdc/kudu.keytab.
I20250903 08:08:14.773751 13095 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUML90r/build/debug/bin/kudu
/tmp/dist-test-taskUML90r/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.201.194:0
--local_ip_for_outbound_sockets=127.12.201.194
--webserver_interface=127.12.201.194
--webserver_port=0
--tserver_master_addrs=127.12.201.254:42407
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.201.212:41681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 08:08:14.879335 17495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:08:14.879549 17495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:08:14.879640 17495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:08:14.882790 17495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 08:08:14.882858 17495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:08:14.883015 17495 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.201.194
I20250903 08:08:14.887203 17495 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.201.212:41681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.201.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.201.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.12.201.254:42407
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17495
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.201.194
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 07:50:08 UTC on 5fd53c4cbb9d
build id 7874
I20250903 08:08:14.888234 17495 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:08:14.889065 17495 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:08:14.895298 17500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:14.895311 17503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:14.895274 17501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:08:14.895569 17495 server_base.cc:1047] running on GCE node
I20250903 08:08:14.895901 17495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:08:14.896402 17495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:08:14.897547 17495 hybrid_clock.cc:648] HybridClock initialized: now 1756886894897533 us; error 28 us; skew 500 ppm
Sep 03 08:08:14 dist-test-slave-27r9 krb5kdc[17247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886894, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 08:08:14.900535 17495 init.cc:377] Logged in from keytab as kudu/127.12.201.194@KRBTEST.COM (short username kudu)
I20250903 08:08:14.901516 17495 webserver.cc:480] Webserver started at http://127.12.201.194:46713/ using document root <none> and password file <none>
I20250903 08:08:14.902046 17495 fs_manager.cc:362] Metadata directory not provided
I20250903 08:08:14.902093 17495 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:08:14.902282 17495 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:08:14.904014 17495 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/minicluster-data/ts-1/data/instance:
uuid: "a5f73f512f944b44919a6d4fdab1837b"
format_stamp: "Formatted at 2025-09-03 08:08:14 on dist-test-slave-27r9"
server_key: "367883a517e6b90d544ffedad4f68e42"
server_key_iv: "bb712eb78981259ee4fcae45545d77c9"
server_key_version: "encryptionkey@0"
I20250903 08:08:14.904459 17495 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/minicluster-data/ts-1/wal/instance:
uuid: "a5f73f512f944b44919a6d4fdab1837b"
format_stamp: "Formatted at 2025-09-03 08:08:14 on dist-test-slave-27r9"
server_key: "367883a517e6b90d544ffedad4f68e42"
server_key_iv: "bb712eb78981259ee4fcae45545d77c9"
server_key_version: "encryptionkey@0"
I20250903 08:08:14.907748 17495 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250903 08:08:14.909792 17510 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:14.910729 17495 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250903 08:08:14.910827 17495 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/minicluster-data/ts-1/wal
uuid: "a5f73f512f944b44919a6d4fdab1837b"
format_stamp: "Formatted at 2025-09-03 08:08:14 on dist-test-slave-27r9"
server_key: "367883a517e6b90d544ffedad4f68e42"
server_key_iv: "bb712eb78981259ee4fcae45545d77c9"
server_key_version: "encryptionkey@0"
I20250903 08:08:14.910910 17495 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:08:14.935305 17495 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:08:14.938308 17495 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:08:14.938488 17495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:08:14.939062 17495 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 08:08:14.939874 17495 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 08:08:14.939922 17495 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:14.939986 17495 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 08:08:14.940016 17495 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:14.948451 17495 rpc_server.cc:307] RPC server started. Bound to: 127.12.201.194:43281
I20250903 08:08:14.948498 17623 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.201.194:43281 every 8 connection(s)
I20250903 08:08:14.949354 17495 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/minicluster-data/ts-1/data/info.pb
Sep 03 08:08:14 dist-test-slave-27r9 krb5kdc[17247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886894, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.194@KRBTEST.COM for kudu/127.12.201.254@KRBTEST.COM
I20250903 08:08:14.959586 13095 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUML90r/build/debug/bin/kudu as pid 17495
I20250903 08:08:14.959695 13095 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/minicluster-data/ts-1/wal/instance
I20250903 08:08:14.959924 13095 external_mini_cluster.cc:1468] Setting key 1c52a98f3dcc93277e65d4f0fedca468
I20250903 08:08:14.960954 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:14.951000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.194:36243 (local address 127.12.201.254:42407)
0903 08:08:14.951133 (+ 133us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:14.951137 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:14.951793 (+ 656us) server_negotiation.cc:408] Connection header received
0903 08:08:14.952640 (+ 847us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:14.952643 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:14.952692 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:14.952764 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:14.954149 (+ 1385us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:14.954678 (+ 529us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:14.955773 (+ 1095us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:14.955904 (+ 131us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:14.957858 (+ 1954us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:14.957876 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:14.957882 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:08:14.957908 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:14.959167 (+ 1259us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:14.959606 (+ 439us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:14.959610 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:14.959612 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:14.959657 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:14.959954 (+ 297us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:14.959958 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:14.959960 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:14.960137 (+ 177us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:14.960230 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:14.960706 (+ 476us) server_negotiation.cc:300] Negotiation successful
0903 08:08:14.960815 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20250903 08:08:14.961735 17626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:14.951237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:42407 (local address 127.12.201.194:36243)
0903 08:08:14.951633 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:14.951683 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:14.952465 (+ 782us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:14.952896 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:14.952903 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:14.953366 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:08:14.953988 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:14.954000 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:14.954779 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:14.954782 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:14.955669 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:14.955677 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:14.955835 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:14.956359 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:08:14.956379 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:08:14.957725 (+ 1346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:08:14.959268 (+ 1543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:14.959272 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:14.959281 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:14.959495 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:14.959750 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:14.959753 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:14.959755 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:14.959862 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:14.960246 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:08:14.960254 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:08:14.960469 (+ 215us) client_negotiation.cc:769] Sending connection context
0903 08:08:14.960682 (+ 213us) client_negotiation.cc:240] Negotiation successful
0903 08:08:14.960973 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":110,"threads_started":1}
I20250903 08:08:14.962965 17624 heartbeater.cc:344] Connected to a master server at 127.12.201.254:42407
I20250903 08:08:14.963281 17624 heartbeater.cc:461] Registering TS with master...
I20250903 08:08:14.963937 17624 heartbeater.cc:507] Master 127.12.201.254:42407 requested a full tablet report, sending...
I20250903 08:08:14.965097 17295 ts_manager.cc:194] Registered new tserver with Master: a5f73f512f944b44919a6d4fdab1837b (127.12.201.194:43281)
I20250903 08:08:14.965662 17295 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.201.194@KRBTEST.COM'} at 127.12.201.194:36243
WARNING: no policy specified for kudu/127.12.201.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.201.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.201.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.201.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.201.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.201.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.201.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.201.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/krb5kdc/kudu.keytab.
I20250903 08:08:15.007543 13095 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUML90r/build/debug/bin/kudu
/tmp/dist-test-taskUML90r/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.201.195:0
--local_ip_for_outbound_sockets=127.12.201.195
--webserver_interface=127.12.201.195
--webserver_port=0
--tserver_master_addrs=127.12.201.254:42407
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.201.212:41681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 08:08:15.110067 17631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:08:15.110273 17631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:08:15.110335 17631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:08:15.113559 17631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 08:08:15.113615 17631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:08:15.113715 17631 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.201.195
I20250903 08:08:15.117828 17631 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.201.212:41681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.201.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.201.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.12.201.254:42407
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17631
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.201.195
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 07:50:08 UTC on 5fd53c4cbb9d
build id 7874
I20250903 08:08:15.118790 17631 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:08:15.119619 17631 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:08:15.125588 17637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:15.125598 17639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:15.125680 17636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:08:15.125881 17631 server_base.cc:1047] running on GCE node
I20250903 08:08:15.126222 17631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:08:15.126721 17631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:08:15.127857 17631 hybrid_clock.cc:648] HybridClock initialized: now 1756886895127841 us; error 35 us; skew 500 ppm
Sep 03 08:08:15 dist-test-slave-27r9 krb5kdc[17247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886895, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 08:08:15.130425 17631 init.cc:377] Logged in from keytab as kudu/127.12.201.195@KRBTEST.COM (short username kudu)
I20250903 08:08:15.131394 17631 webserver.cc:480] Webserver started at http://127.12.201.195:43789/ using document root <none> and password file <none>
I20250903 08:08:15.131899 17631 fs_manager.cc:362] Metadata directory not provided
I20250903 08:08:15.131959 17631 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:08:15.132159 17631 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:08:15.133797 17631 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/minicluster-data/ts-2/data/instance:
uuid: "139d1667d01f4396835f6f5c353e9937"
format_stamp: "Formatted at 2025-09-03 08:08:15 on dist-test-slave-27r9"
server_key: "44931eb640c086dd3c14010dcaab7ba0"
server_key_iv: "f76965636654c6a8b046a4a8e452d4cf"
server_key_version: "encryptionkey@0"
I20250903 08:08:15.134261 17631 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/minicluster-data/ts-2/wal/instance:
uuid: "139d1667d01f4396835f6f5c353e9937"
format_stamp: "Formatted at 2025-09-03 08:08:15 on dist-test-slave-27r9"
server_key: "44931eb640c086dd3c14010dcaab7ba0"
server_key_iv: "f76965636654c6a8b046a4a8e452d4cf"
server_key_version: "encryptionkey@0"
I20250903 08:08:15.137446 17631 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250903 08:08:15.139540 17646 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:15.140523 17631 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250903 08:08:15.140625 17631 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/minicluster-data/ts-2/wal
uuid: "139d1667d01f4396835f6f5c353e9937"
format_stamp: "Formatted at 2025-09-03 08:08:15 on dist-test-slave-27r9"
server_key: "44931eb640c086dd3c14010dcaab7ba0"
server_key_iv: "f76965636654c6a8b046a4a8e452d4cf"
server_key_version: "encryptionkey@0"
I20250903 08:08:15.140713 17631 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:08:15.150295 17631 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:08:15.153159 17631 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:08:15.153321 17631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:08:15.153839 17631 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 08:08:15.154597 17631 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 08:08:15.154645 17631 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:15.154711 17631 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 08:08:15.154742 17631 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:15.163321 17631 rpc_server.cc:307] RPC server started. Bound to: 127.12.201.195:39455
I20250903 08:08:15.163349 17759 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.201.195:39455 every 8 connection(s)
I20250903 08:08:15.164232 17631 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/minicluster-data/ts-2/data/info.pb
Sep 03 08:08:15 dist-test-slave-27r9 krb5kdc[17247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886895, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.195@KRBTEST.COM for kudu/127.12.201.254@KRBTEST.COM
I20250903 08:08:15.172839 13095 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUML90r/build/debug/bin/kudu as pid 17631
I20250903 08:08:15.172982 13095 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756886860253814-13095-0/minicluster-data/ts-2/wal/instance
I20250903 08:08:15.173238 13095 external_mini_cluster.cc:1468] Setting key 6eb9349c6aeaacf7163e2b27e081518a
I20250903 08:08:15.176231 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:15.165885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.195:33933 (local address 127.12.201.254:42407)
0903 08:08:15.166014 (+ 129us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:15.166018 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:15.166603 (+ 585us) server_negotiation.cc:408] Connection header received
0903 08:08:15.167443 (+ 840us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:15.167446 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:15.167497 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:15.167566 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:15.168945 (+ 1379us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:15.169432 (+ 487us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:15.170588 (+ 1156us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:15.170751 (+ 163us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:15.172773 (+ 2022us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:15.172796 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:15.172800 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:08:15.172833 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:15.174555 (+ 1722us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:15.175008 (+ 453us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:15.175012 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:15.175014 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:15.175063 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:15.175372 (+ 309us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:15.175375 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:15.175376 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:15.175513 (+ 137us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:15.175588 (+ 75us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:15.176029 (+ 441us) server_negotiation.cc:300] Negotiation successful
0903 08:08:15.176117 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20250903 08:08:15.176869 17762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:15.166058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:42407 (local address 127.12.201.195:33933)
0903 08:08:15.166456 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:15.166492 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:15.167262 (+ 770us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:15.167717 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:15.167724 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:15.168098 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:08:15.168790 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:15.168805 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:15.169588 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:15.169592 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:15.170455 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:15.170463 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:15.170619 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:15.171210 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:08:15.171231 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:08:15.172603 (+ 1372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:08:15.174660 (+ 2057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:15.174664 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:15.174673 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:15.174896 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:15.175164 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:15.175170 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:15.175173 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:15.175277 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:15.175602 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:08:15.175607 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:08:15.175816 (+ 209us) client_negotiation.cc:769] Sending connection context
0903 08:08:15.175986 (+ 170us) client_negotiation.cc:240] Negotiation successful
0903 08:08:15.176186 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":106,"threads_started":1}
I20250903 08:08:15.177870 17760 heartbeater.cc:344] Connected to a master server at 127.12.201.254:42407
I20250903 08:08:15.178085 17760 heartbeater.cc:461] Registering TS with master...
I20250903 08:08:15.178542 17760 heartbeater.cc:507] Master 127.12.201.254:42407 requested a full tablet report, sending...
I20250903 08:08:15.179466 17295 ts_manager.cc:194] Registered new tserver with Master: 139d1667d01f4396835f6f5c353e9937 (127.12.201.195:39455)
I20250903 08:08:15.179986 17295 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.201.195@KRBTEST.COM'} at 127.12.201.195:33933
I20250903 08:08:15.187062 13095 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Sep 03 08:08:15 dist-test-slave-27r9 krb5kdc[17247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 03 08:08:15 dist-test-slave-27r9 krb5kdc[17247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.201.193@KRBTEST.COM
I20250903 08:08:15.209748 17767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:15.201343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44616 (local address 127.12.201.193:43541)
0903 08:08:15.201633 (+ 290us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:15.201639 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:15.201661 (+ 22us) server_negotiation.cc:408] Connection header received
0903 08:08:15.201720 (+ 59us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:15.201725 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:15.201883 (+ 158us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:15.202013 (+ 130us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:15.202731 (+ 718us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:15.203483 (+ 752us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:15.204161 (+ 678us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:15.204324 (+ 163us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:15.206516 (+ 2192us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:15.206537 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:15.206548 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:08:15.206571 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:15.208303 (+ 1732us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:15.208709 (+ 406us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:15.208714 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:15.208716 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:15.208764 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:15.208952 (+ 188us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:15.208954 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:15.208955 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:15.209117 (+ 162us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:15.209257 (+ 140us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:15.209398 (+ 141us) server_negotiation.cc:300] Negotiation successful
0903 08:08:15.209524 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":114,"threads_started":1}
Sep 03 08:08:15 dist-test-slave-27r9 krb5kdc[17247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.201.254@KRBTEST.COM
I20250903 08:08:15.219223 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:15.212385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46302 (local address 127.12.201.254:42407)
0903 08:08:15.212550 (+ 165us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:15.212555 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:15.212568 (+ 13us) server_negotiation.cc:408] Connection header received
0903 08:08:15.212600 (+ 32us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:15.212602 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:15.212640 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:15.212721 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:15.213396 (+ 675us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:15.213879 (+ 483us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:15.214432 (+ 553us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:15.214646 (+ 214us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:15.216584 (+ 1938us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:15.216600 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:15.216602 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:08:15.216624 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:15.217787 (+ 1163us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:15.218175 (+ 388us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:15.218178 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:15.218180 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:15.218219 (+ 39us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:15.218479 (+ 260us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:15.218482 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:15.218483 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:15.218652 (+ 169us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:15.218758 (+ 106us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:15.218907 (+ 149us) server_negotiation.cc:300] Negotiation successful
0903 08:08:15.219040 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20250903 08:08:15.219571 17295 server_base.cc:1129] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46302
I20250903 08:08:15.220145 13095 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUML90r/build/debug/bin/kudu with pid 17359
I20250903 08:08:15.225517 13095 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUML90r/build/debug/bin/kudu with pid 17495
I20250903 08:08:15.230691 13095 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUML90r/build/debug/bin/kudu with pid 17631
I20250903 08:08:15.235873 13095 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUML90r/build/debug/bin/kudu with pid 17263
2025-09-03T08:08:15Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3072 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 03 08:08:15 dist-test-slave-27r9 krb5kdc[17772](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 03 08:08:15 dist-test-slave-27r9 krb5kdc[17772](info): set up 2 sockets
Sep 03 08:08:15 dist-test-slave-27r9 krb5kdc[17772](info): commencing operation
krb5kdc: starting...
W20250903 08:08:17.227401 13095 mini_kdc.cc:121] Time spent starting KDC: real 1.970s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/krb5kdc/test-user.keytab.
Sep 03 08:08:17 dist-test-slave-27r9 krb5kdc[17772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886897, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-09-03T08:08:17Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-03T08:08:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.201.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.201.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.201.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.201.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.201.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.201.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.201.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.201.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/krb5kdc/kudu.keytab.
I20250903 08:08:17.367615 13095 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUML90r/build/debug/bin/kudu
/tmp/dist-test-taskUML90r/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.201.254:37865
--webserver_interface=127.12.201.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.201.212:41211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.201.254:37865
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 08:08:17.470187 17788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:08:17.470436 17788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:08:17.470537 17788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:08:17.473776 17788 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250903 08:08:17.473843 17788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 08:08:17.473876 17788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:08:17.473907 17788 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250903 08:08:17.473939 17788 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250903 08:08:17.478266 17788 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.201.212:41211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.201.254:37865
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.201.254:37865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.201.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17788
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 07:50:08 UTC on 5fd53c4cbb9d
build id 7874
I20250903 08:08:17.479399 17788 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:08:17.480253 17788 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:08:17.485442 17793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:17.485445 17794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:17.485442 17796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:08:17.485591 17788 server_base.cc:1047] running on GCE node
I20250903 08:08:17.486167 17788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:08:17.487064 17788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:08:17.488224 17788 hybrid_clock.cc:648] HybridClock initialized: now 1756886897488214 us; error 28 us; skew 500 ppm
Sep 03 08:08:17 dist-test-slave-27r9 krb5kdc[17772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886897, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 08:08:17.490890 17788 init.cc:377] Logged in from keytab as kudu/127.12.201.254@KRBTEST.COM (short username kudu)
I20250903 08:08:17.491586 17788 fs_manager.cc:362] Metadata directory not provided
I20250903 08:08:17.491644 17788 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:08:17.491829 17788 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:08:17.493571 17788 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/minicluster-data/master-0/data/instance:
uuid: "b18a2a441e5b49a0ac165065b5fbf71b"
format_stamp: "Formatted at 2025-09-03 08:08:17 on dist-test-slave-27r9"
server_key: "c2d88d8ceb1d5b89f033277b829eb661"
server_key_iv: "700cd7c1d4b67ec6bcf6b3810f0bd884"
server_key_version: "encryptionkey@0"
I20250903 08:08:17.494122 17788 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/minicluster-data/master-0/wal/instance:
uuid: "b18a2a441e5b49a0ac165065b5fbf71b"
format_stamp: "Formatted at 2025-09-03 08:08:17 on dist-test-slave-27r9"
server_key: "c2d88d8ceb1d5b89f033277b829eb661"
server_key_iv: "700cd7c1d4b67ec6bcf6b3810f0bd884"
server_key_version: "encryptionkey@0"
I20250903 08:08:17.497527 17788 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20250903 08:08:17.499778 17802 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:17.500773 17788 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250903 08:08:17.500890 17788 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/minicluster-data/master-0/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/minicluster-data/master-0/wal
uuid: "b18a2a441e5b49a0ac165065b5fbf71b"
format_stamp: "Formatted at 2025-09-03 08:08:17 on dist-test-slave-27r9"
server_key: "c2d88d8ceb1d5b89f033277b829eb661"
server_key_iv: "700cd7c1d4b67ec6bcf6b3810f0bd884"
server_key_version: "encryptionkey@0"
I20250903 08:08:17.500974 17788 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:08:17.527050 17788 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:08:17.534374 17788 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:08:17.534566 17788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:08:17.541774 17788 rpc_server.cc:307] RPC server started. Bound to: 127.12.201.254:37865
I20250903 08:08:17.541757 17854 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.201.254:37865 every 8 connection(s)
I20250903 08:08:17.542444 17788 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/minicluster-data/master-0/data/info.pb
I20250903 08:08:17.543006 13095 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUML90r/build/debug/bin/kudu as pid 17788
I20250903 08:08:17.543123 13095 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/minicluster-data/master-0/wal/instance
I20250903 08:08:17.543357 13095 external_mini_cluster.cc:1468] Setting key e8f2a7a6c13771a3da190d51a8b49c4b
I20250903 08:08:17.545385 17855 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:08:17.551416 17855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b18a2a441e5b49a0ac165065b5fbf71b: Bootstrap starting.
Sep 03 08:08:17 dist-test-slave-27r9 krb5kdc[17772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886897, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.201.254@KRBTEST.COM
I20250903 08:08:17.554010 17855 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b18a2a441e5b49a0ac165065b5fbf71b: Neither blocks nor log segments found. Creating new log.
I20250903 08:08:17.554881 17855 log.cc:826] T 00000000000000000000000000000000 P b18a2a441e5b49a0ac165065b5fbf71b: Log is configured to *not* fsync() on all Append() calls
I20250903 08:08:17.557061 17855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b18a2a441e5b49a0ac165065b5fbf71b: No bootstrap required, opened a new log
I20250903 08:08:17.557636 17858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:17.544703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40234 (local address 127.12.201.254:37865)
0903 08:08:17.545094 (+ 391us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:17.545107 (+ 13us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:17.545150 (+ 43us) server_negotiation.cc:408] Connection header received
0903 08:08:17.546549 (+ 1399us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:17.546586 (+ 37us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:17.546959 (+ 373us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:17.547345 (+ 386us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:17.548186 (+ 841us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:17.549323 (+ 1137us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:17.550067 (+ 744us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:17.550373 (+ 306us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:17.552640 (+ 2267us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:17.552688 (+ 48us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:17.552705 (+ 17us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:08:17.552747 (+ 42us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:17.554814 (+ 2067us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:17.555275 (+ 461us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:17.555281 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:17.555287 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:17.555366 (+ 79us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:17.555703 (+ 337us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:17.555707 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:17.555709 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:17.556122 (+ 413us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:17.556359 (+ 237us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:17.556566 (+ 207us) server_negotiation.cc:300] Negotiation successful
0903 08:08:17.556841 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":101,"threads_started":1}
I20250903 08:08:17.559903 17855 raft_consensus.cc:357] T 00000000000000000000000000000000 P b18a2a441e5b49a0ac165065b5fbf71b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b18a2a441e5b49a0ac165065b5fbf71b" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 37865 } }
I20250903 08:08:17.560070 17855 raft_consensus.cc:383] T 00000000000000000000000000000000 P b18a2a441e5b49a0ac165065b5fbf71b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:08:17.560104 17855 raft_consensus.cc:738] T 00000000000000000000000000000000 P b18a2a441e5b49a0ac165065b5fbf71b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b18a2a441e5b49a0ac165065b5fbf71b, State: Initialized, Role: FOLLOWER
I20250903 08:08:17.560477 17855 consensus_queue.cc:260] T 00000000000000000000000000000000 P b18a2a441e5b49a0ac165065b5fbf71b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b18a2a441e5b49a0ac165065b5fbf71b" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 37865 } }
I20250903 08:08:17.560590 17855 raft_consensus.cc:397] T 00000000000000000000000000000000 P b18a2a441e5b49a0ac165065b5fbf71b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250903 08:08:17.560635 17855 raft_consensus.cc:491] T 00000000000000000000000000000000 P b18a2a441e5b49a0ac165065b5fbf71b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250903 08:08:17.560715 17855 raft_consensus.cc:3058] T 00000000000000000000000000000000 P b18a2a441e5b49a0ac165065b5fbf71b [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:08:17.561614 17855 raft_consensus.cc:513] T 00000000000000000000000000000000 P b18a2a441e5b49a0ac165065b5fbf71b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b18a2a441e5b49a0ac165065b5fbf71b" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 37865 } }
I20250903 08:08:17.561923 17855 leader_election.cc:304] T 00000000000000000000000000000000 P b18a2a441e5b49a0ac165065b5fbf71b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b18a2a441e5b49a0ac165065b5fbf71b; no voters:
I20250903 08:08:17.562191 17855 leader_election.cc:290] T 00000000000000000000000000000000 P b18a2a441e5b49a0ac165065b5fbf71b [CANDIDATE]: Term 1 election: Requested vote from peers
I20250903 08:08:17.562330 17860 raft_consensus.cc:2802] T 00000000000000000000000000000000 P b18a2a441e5b49a0ac165065b5fbf71b [term 1 FOLLOWER]: Leader election won for term 1
I20250903 08:08:17.562536 17860 raft_consensus.cc:695] T 00000000000000000000000000000000 P b18a2a441e5b49a0ac165065b5fbf71b [term 1 LEADER]: Becoming Leader. State: Replica: b18a2a441e5b49a0ac165065b5fbf71b, State: Running, Role: LEADER
I20250903 08:08:17.562819 17860 consensus_queue.cc:237] T 00000000000000000000000000000000 P b18a2a441e5b49a0ac165065b5fbf71b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b18a2a441e5b49a0ac165065b5fbf71b" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 37865 } }
I20250903 08:08:17.563210 17855 sys_catalog.cc:564] T 00000000000000000000000000000000 P b18a2a441e5b49a0ac165065b5fbf71b [sys.catalog]: configured and running, proceeding with master startup.
I20250903 08:08:17.564373 17861 sys_catalog.cc:455] T 00000000000000000000000000000000 P b18a2a441e5b49a0ac165065b5fbf71b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b18a2a441e5b49a0ac165065b5fbf71b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b18a2a441e5b49a0ac165065b5fbf71b" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 37865 } } }
I20250903 08:08:17.564386 17862 sys_catalog.cc:455] T 00000000000000000000000000000000 P b18a2a441e5b49a0ac165065b5fbf71b [sys.catalog]: SysCatalogTable state changed. Reason: New leader b18a2a441e5b49a0ac165065b5fbf71b. Latest consensus state: current_term: 1 leader_uuid: "b18a2a441e5b49a0ac165065b5fbf71b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b18a2a441e5b49a0ac165065b5fbf71b" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 37865 } } }
I20250903 08:08:17.564635 17862 sys_catalog.cc:458] T 00000000000000000000000000000000 P b18a2a441e5b49a0ac165065b5fbf71b [sys.catalog]: This master's current role is: LEADER
I20250903 08:08:17.564594 17861 sys_catalog.cc:458] T 00000000000000000000000000000000 P b18a2a441e5b49a0ac165065b5fbf71b [sys.catalog]: This master's current role is: LEADER
I20250903 08:08:17.565510 17869 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250903 08:08:17.568013 17869 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250903 08:08:17.572681 17869 catalog_manager.cc:1349] Generated new cluster ID: 2d092157e7f145818b94d2f9aa346ef3
I20250903 08:08:17.572747 17869 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250903 08:08:17.585542 17869 catalog_manager.cc:1372] Generated new certificate authority record
I20250903 08:08:17.586375 17869 catalog_manager.cc:1506] Loading token signing keys...
I20250903 08:08:17.592502 17869 catalog_manager.cc:5955] T 00000000000000000000000000000000 P b18a2a441e5b49a0ac165065b5fbf71b: Generated new TSK 0
I20250903 08:08:17.593258 17869 catalog_manager.cc:1516] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.201.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.201.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.201.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.201.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.201.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.201.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.201.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.201.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/krb5kdc/kudu.keytab.
I20250903 08:08:17.650208 13095 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUML90r/build/debug/bin/kudu
/tmp/dist-test-taskUML90r/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.201.193:0
--local_ip_for_outbound_sockets=127.12.201.193
--webserver_interface=127.12.201.193
--webserver_port=0
--tserver_master_addrs=127.12.201.254:37865
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.201.212:41211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 08:08:17.757647 17883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:08:17.757845 17883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:08:17.757920 17883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:08:17.761279 17883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 08:08:17.761350 17883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:08:17.761560 17883 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.201.193
I20250903 08:08:17.766026 17883 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.201.212:41211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.201.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.201.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.12.201.254:37865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17883
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.201.193
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 07:50:08 UTC on 5fd53c4cbb9d
build id 7874
I20250903 08:08:17.767096 17883 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:08:17.767963 17883 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:08:17.774001 17891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:17.774085 17889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:17.774098 17888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:08:17.774240 17883 server_base.cc:1047] running on GCE node
I20250903 08:08:17.774703 17883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:08:17.775331 17883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:08:17.776499 17883 hybrid_clock.cc:648] HybridClock initialized: now 1756886897776480 us; error 33 us; skew 500 ppm
Sep 03 08:08:17 dist-test-slave-27r9 krb5kdc[17772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886897, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 08:08:17.779146 17883 init.cc:377] Logged in from keytab as kudu/127.12.201.193@KRBTEST.COM (short username kudu)
I20250903 08:08:17.779795 17883 fs_manager.cc:362] Metadata directory not provided
I20250903 08:08:17.779848 17883 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:08:17.779997 17883 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:08:17.781694 17883 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/minicluster-data/ts-0/data/instance:
uuid: "51b1391323444680bed816270d464dc2"
format_stamp: "Formatted at 2025-09-03 08:08:17 on dist-test-slave-27r9"
server_key: "86ee340cf784df6fd987beeb0a6fc966"
server_key_iv: "a4c0b6a8e8c449f5fe10f6a5f13cb23a"
server_key_version: "encryptionkey@0"
I20250903 08:08:17.782104 17883 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/minicluster-data/ts-0/wal/instance:
uuid: "51b1391323444680bed816270d464dc2"
format_stamp: "Formatted at 2025-09-03 08:08:17 on dist-test-slave-27r9"
server_key: "86ee340cf784df6fd987beeb0a6fc966"
server_key_iv: "a4c0b6a8e8c449f5fe10f6a5f13cb23a"
server_key_version: "encryptionkey@0"
I20250903 08:08:17.785168 17883 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250903 08:08:17.787204 17897 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:17.788173 17883 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250903 08:08:17.788264 17883 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/minicluster-data/ts-0/wal
uuid: "51b1391323444680bed816270d464dc2"
format_stamp: "Formatted at 2025-09-03 08:08:17 on dist-test-slave-27r9"
server_key: "86ee340cf784df6fd987beeb0a6fc966"
server_key_iv: "a4c0b6a8e8c449f5fe10f6a5f13cb23a"
server_key_version: "encryptionkey@0"
I20250903 08:08:17.788373 17883 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:08:17.804186 17883 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:08:17.807439 17883 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:08:17.807663 17883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:08:17.808235 17883 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 08:08:17.809089 17883 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 08:08:17.809136 17883 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:17.809185 17883 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 08:08:17.809218 17883 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:17.817876 17883 rpc_server.cc:307] RPC server started. Bound to: 127.12.201.193:46185
I20250903 08:08:17.817917 18010 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.201.193:46185 every 8 connection(s)
I20250903 08:08:17.818434 17883 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/minicluster-data/ts-0/data/info.pb
Sep 03 08:08:17 dist-test-slave-27r9 krb5kdc[17772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886897, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.193@KRBTEST.COM for kudu/127.12.201.254@KRBTEST.COM
I20250903 08:08:17.826085 13095 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUML90r/build/debug/bin/kudu as pid 17883
I20250903 08:08:17.826197 13095 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/minicluster-data/ts-0/wal/instance
I20250903 08:08:17.826463 13095 external_mini_cluster.cc:1468] Setting key acc41e26ddaef545f3ad94c12045e34c
I20250903 08:08:17.830717 17858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:17.820106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.193:51403 (local address 127.12.201.254:37865)
0903 08:08:17.820286 (+ 180us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:17.820292 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:17.820852 (+ 560us) server_negotiation.cc:408] Connection header received
0903 08:08:17.821617 (+ 765us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:17.821620 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:17.821668 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:17.821739 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:17.823364 (+ 1625us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:17.823890 (+ 526us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:17.824473 (+ 583us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:17.824593 (+ 120us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:17.826862 (+ 2269us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:17.826886 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:17.826890 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:08:17.826919 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:17.828548 (+ 1629us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:17.829204 (+ 656us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:17.829208 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:17.829210 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:17.829259 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:17.829637 (+ 378us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:17.829642 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:17.829644 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:17.829807 (+ 163us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:17.829867 (+ 60us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:17.830409 (+ 542us) server_negotiation.cc:300] Negotiation successful
0903 08:08:17.830537 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20250903 08:08:17.831451 18013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:17.820324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:37865 (local address 127.12.201.193:51403)
0903 08:08:17.820716 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:17.820751 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:17.821443 (+ 692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:17.821895 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:17.821903 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:17.822282 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:08:17.823234 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:17.823249 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:17.823992 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:17.823995 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:17.824381 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:17.824386 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:17.824532 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:17.825121 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:08:17.825145 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:08:17.826688 (+ 1543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:08:17.828687 (+ 1999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:17.828696 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:17.828712 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:17.829075 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:17.829362 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:17.829368 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:17.829373 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:17.829501 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:17.829946 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:08:17.829950 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:08:17.830161 (+ 211us) client_negotiation.cc:769] Sending connection context
0903 08:08:17.830346 (+ 185us) client_negotiation.cc:240] Negotiation successful
0903 08:08:17.830622 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":93,"threads_started":1}
I20250903 08:08:17.832719 18011 heartbeater.cc:344] Connected to a master server at 127.12.201.254:37865
I20250903 08:08:17.832962 18011 heartbeater.cc:461] Registering TS with master...
I20250903 08:08:17.833431 18011 heartbeater.cc:507] Master 127.12.201.254:37865 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.12.201.194@KRBTEST.COM; defaulting to no policy
I20250903 08:08:17.835111 17819 ts_manager.cc:194] Registered new tserver with Master: 51b1391323444680bed816270d464dc2 (127.12.201.193:46185)
I20250903 08:08:17.836359 17819 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.201.193@KRBTEST.COM'} at 127.12.201.193:51403
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.201.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.201.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.201.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.201.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.201.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.201.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.201.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/krb5kdc/kudu.keytab.
I20250903 08:08:17.877941 13095 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUML90r/build/debug/bin/kudu
/tmp/dist-test-taskUML90r/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.201.194:0
--local_ip_for_outbound_sockets=127.12.201.194
--webserver_interface=127.12.201.194
--webserver_port=0
--tserver_master_addrs=127.12.201.254:37865
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.201.212:41211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 08:08:17.982249 18018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:08:17.982434 18018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:08:17.982491 18018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:08:17.985613 18018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 08:08:17.985679 18018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:08:17.985755 18018 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.201.194
I20250903 08:08:17.989866 18018 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.201.212:41211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.201.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.201.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.12.201.254:37865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18018
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.201.194
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 07:50:08 UTC on 5fd53c4cbb9d
build id 7874
I20250903 08:08:17.990844 18018 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:08:17.991729 18018 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:08:17.998049 18026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:17.998070 18023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:17.998092 18024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:08:17.998185 18018 server_base.cc:1047] running on GCE node
I20250903 08:08:17.998705 18018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:08:17.999291 18018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:08:18.000463 18018 hybrid_clock.cc:648] HybridClock initialized: now 1756886898000434 us; error 41 us; skew 500 ppm
Sep 03 08:08:17 dist-test-slave-27r9 krb5kdc[17772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886897, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 08:08:18.003260 18018 init.cc:377] Logged in from keytab as kudu/127.12.201.194@KRBTEST.COM (short username kudu)
I20250903 08:08:18.003958 18018 fs_manager.cc:362] Metadata directory not provided
I20250903 08:08:18.004021 18018 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:08:18.004204 18018 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:08:18.005982 18018 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/minicluster-data/ts-1/data/instance:
uuid: "76674bec22a54f67aa16f1e68af86408"
format_stamp: "Formatted at 2025-09-03 08:08:18 on dist-test-slave-27r9"
server_key: "c4f81958a96cd52dbde04afc30417c33"
server_key_iv: "18b972e9181846e57b39c76bae68faa2"
server_key_version: "encryptionkey@0"
I20250903 08:08:18.006461 18018 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/minicluster-data/ts-1/wal/instance:
uuid: "76674bec22a54f67aa16f1e68af86408"
format_stamp: "Formatted at 2025-09-03 08:08:18 on dist-test-slave-27r9"
server_key: "c4f81958a96cd52dbde04afc30417c33"
server_key_iv: "18b972e9181846e57b39c76bae68faa2"
server_key_version: "encryptionkey@0"
I20250903 08:08:18.009795 18018 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250903 08:08:18.012004 18032 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:18.012996 18018 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250903 08:08:18.013098 18018 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/minicluster-data/ts-1/wal
uuid: "76674bec22a54f67aa16f1e68af86408"
format_stamp: "Formatted at 2025-09-03 08:08:18 on dist-test-slave-27r9"
server_key: "c4f81958a96cd52dbde04afc30417c33"
server_key_iv: "18b972e9181846e57b39c76bae68faa2"
server_key_version: "encryptionkey@0"
I20250903 08:08:18.013187 18018 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:08:18.030681 18018 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:08:18.033584 18018 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:08:18.033764 18018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:08:18.034322 18018 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 08:08:18.035130 18018 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 08:08:18.035176 18018 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:18.035233 18018 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 08:08:18.035281 18018 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:18.043445 18018 rpc_server.cc:307] RPC server started. Bound to: 127.12.201.194:38235
I20250903 08:08:18.043488 18145 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.201.194:38235 every 8 connection(s)
I20250903 08:08:18.043969 18018 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/minicluster-data/ts-1/data/info.pb
Sep 03 08:08:18 dist-test-slave-27r9 krb5kdc[17772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886897, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.194@KRBTEST.COM for kudu/127.12.201.254@KRBTEST.COM
I20250903 08:08:18.053380 13095 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUML90r/build/debug/bin/kudu as pid 18018
I20250903 08:08:18.053510 13095 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/minicluster-data/ts-1/wal/instance
I20250903 08:08:18.053797 13095 external_mini_cluster.cc:1468] Setting key eed233728346ff0797ca60d61a6b5619
I20250903 08:08:18.055940 17858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:18.045586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.194:60139 (local address 127.12.201.254:37865)
0903 08:08:18.045740 (+ 154us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:18.045744 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:18.046332 (+ 588us) server_negotiation.cc:408] Connection header received
0903 08:08:18.047166 (+ 834us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:18.047170 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:18.047251 (+ 81us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:18.047332 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:18.048989 (+ 1657us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:18.049486 (+ 497us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:18.050134 (+ 648us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:18.050315 (+ 181us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:18.052444 (+ 2129us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:18.052463 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:18.052465 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:08:18.052490 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:18.054012 (+ 1522us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:18.054490 (+ 478us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:18.054494 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:18.054495 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:18.054553 (+ 58us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:18.054815 (+ 262us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:18.054818 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:18.054820 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:18.055015 (+ 195us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:18.055117 (+ 102us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:18.055657 (+ 540us) server_negotiation.cc:300] Negotiation successful
0903 08:08:18.055769 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20250903 08:08:18.056787 18148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:18.045816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:37865 (local address 127.12.201.194:60139)
0903 08:08:18.046204 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:18.046238 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:18.046941 (+ 703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:18.047505 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:18.047512 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:18.047914 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:08:18.048824 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:18.048837 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:18.049614 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:18.049617 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:18.050005 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:18.050012 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:18.050204 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:18.050836 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:08:18.050857 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:08:18.052299 (+ 1442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:08:18.054122 (+ 1823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:18.054127 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:18.054136 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:18.054389 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:18.054642 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:18.054644 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:18.054646 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:18.054729 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:18.055105 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:08:18.055111 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:08:18.055360 (+ 249us) client_negotiation.cc:769] Sending connection context
0903 08:08:18.055594 (+ 234us) client_negotiation.cc:240] Negotiation successful
0903 08:08:18.055912 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":95,"threads_started":1}
I20250903 08:08:18.058498 18146 heartbeater.cc:344] Connected to a master server at 127.12.201.254:37865
I20250903 08:08:18.058822 18146 heartbeater.cc:461] Registering TS with master...
I20250903 08:08:18.059437 18146 heartbeater.cc:507] Master 127.12.201.254:37865 requested a full tablet report, sending...
I20250903 08:08:18.060854 17819 ts_manager.cc:194] Registered new tserver with Master: 76674bec22a54f67aa16f1e68af86408 (127.12.201.194:38235)
I20250903 08:08:18.061599 17819 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.201.194@KRBTEST.COM'} at 127.12.201.194:60139
WARNING: no policy specified for kudu/127.12.201.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.201.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.201.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.201.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.201.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.201.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.201.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.201.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/krb5kdc/kudu.keytab.
I20250903 08:08:18.101959 13095 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUML90r/build/debug/bin/kudu
/tmp/dist-test-taskUML90r/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.201.195:0
--local_ip_for_outbound_sockets=127.12.201.195
--webserver_interface=127.12.201.195
--webserver_port=0
--tserver_master_addrs=127.12.201.254:37865
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.201.212:41211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 08:08:18.206411 18153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:08:18.206617 18153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:08:18.206733 18153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:08:18.210466 18153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 08:08:18.210541 18153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:08:18.210649 18153 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.201.195
I20250903 08:08:18.214849 18153 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.201.212:41211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.201.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.201.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.12.201.254:37865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18153
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.201.195
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 07:50:08 UTC on 5fd53c4cbb9d
build id 7874
I20250903 08:08:18.215907 18153 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:08:18.216760 18153 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:08:18.222718 18159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:18.222739 18158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:08:18.222924 18153 server_base.cc:1047] running on GCE node
W20250903 08:08:18.222827 18161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:08:18.223385 18153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:08:18.223933 18153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:08:18.225100 18153 hybrid_clock.cc:648] HybridClock initialized: now 1756886898225083 us; error 29 us; skew 500 ppm
Sep 03 08:08:18 dist-test-slave-27r9 krb5kdc[17772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886898, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 08:08:18.227921 18153 init.cc:377] Logged in from keytab as kudu/127.12.201.195@KRBTEST.COM (short username kudu)
I20250903 08:08:18.228633 18153 fs_manager.cc:362] Metadata directory not provided
I20250903 08:08:18.228696 18153 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:08:18.228881 18153 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:08:18.230581 18153 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/minicluster-data/ts-2/data/instance:
uuid: "f6ad3857e5914ab8aca359575d246d4a"
format_stamp: "Formatted at 2025-09-03 08:08:18 on dist-test-slave-27r9"
server_key: "dcaf5a94b345f0cf7966c5bbe4f8f5a6"
server_key_iv: "cc12b0d98f61c9334c6e06803263d606"
server_key_version: "encryptionkey@0"
I20250903 08:08:18.231099 18153 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/minicluster-data/ts-2/wal/instance:
uuid: "f6ad3857e5914ab8aca359575d246d4a"
format_stamp: "Formatted at 2025-09-03 08:08:18 on dist-test-slave-27r9"
server_key: "dcaf5a94b345f0cf7966c5bbe4f8f5a6"
server_key_iv: "cc12b0d98f61c9334c6e06803263d606"
server_key_version: "encryptionkey@0"
I20250903 08:08:18.234341 18153 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20250903 08:08:18.236537 18167 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:18.237473 18153 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250903 08:08:18.237574 18153 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/minicluster-data/ts-2/wal
uuid: "f6ad3857e5914ab8aca359575d246d4a"
format_stamp: "Formatted at 2025-09-03 08:08:18 on dist-test-slave-27r9"
server_key: "dcaf5a94b345f0cf7966c5bbe4f8f5a6"
server_key_iv: "cc12b0d98f61c9334c6e06803263d606"
server_key_version: "encryptionkey@0"
I20250903 08:08:18.237664 18153 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:08:18.257459 18153 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:08:18.260207 18153 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:08:18.260383 18153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:08:18.260943 18153 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 08:08:18.261754 18153 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 08:08:18.261801 18153 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:18.261857 18153 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 08:08:18.261889 18153 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:18.270812 18153 rpc_server.cc:307] RPC server started. Bound to: 127.12.201.195:32805
I20250903 08:08:18.270874 18280 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.201.195:32805 every 8 connection(s)
I20250903 08:08:18.271428 18153 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/minicluster-data/ts-2/data/info.pb
I20250903 08:08:18.277206 13095 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUML90r/build/debug/bin/kudu as pid 18153
I20250903 08:08:18.277308 13095 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/minicluster-data/ts-2/wal/instance
I20250903 08:08:18.277524 13095 external_mini_cluster.cc:1468] Setting key f68570be996fdae5534cef91ced2df8c
Sep 03 08:08:18 dist-test-slave-27r9 krb5kdc[17772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886898, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.195@KRBTEST.COM for kudu/127.12.201.254@KRBTEST.COM
I20250903 08:08:18.282783 17858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:18.273039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.195:45683 (local address 127.12.201.254:37865)
0903 08:08:18.273188 (+ 149us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:18.273192 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:18.273838 (+ 646us) server_negotiation.cc:408] Connection header received
0903 08:08:18.274605 (+ 767us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:18.274608 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:18.274655 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:18.274732 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:18.276285 (+ 1553us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:18.276787 (+ 502us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:18.277464 (+ 677us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:18.277647 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:18.280048 (+ 2401us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:18.280062 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:18.280064 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:08:18.280088 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:18.281327 (+ 1239us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:18.281733 (+ 406us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:18.281735 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:18.281737 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:18.281774 (+ 37us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:18.281987 (+ 213us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:18.281989 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:18.281990 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:18.282122 (+ 132us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:18.282193 (+ 71us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:18.282580 (+ 387us) server_negotiation.cc:300] Negotiation successful
0903 08:08:18.282669 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20250903 08:08:18.283453 18283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:18.273299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:37865 (local address 127.12.201.195:45683)
0903 08:08:18.273695 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:18.273730 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:18.274429 (+ 699us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:18.274868 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:18.274876 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:18.275258 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:08:18.276158 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:18.276171 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:18.276904 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:18.276907 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:18.277346 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:18.277357 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:18.277544 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:18.278215 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:08:18.278247 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:08:18.279914 (+ 1667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:08:18.281424 (+ 1510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:18.281428 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:18.281436 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:18.281650 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:18.281842 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:18.281844 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:18.281846 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:18.281917 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:18.282192 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:08:18.282196 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:08:18.282389 (+ 193us) client_negotiation.cc:769] Sending connection context
0903 08:08:18.282546 (+ 157us) client_negotiation.cc:240] Negotiation successful
0903 08:08:18.282740 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":97,"threads_started":1}
I20250903 08:08:18.284435 18281 heartbeater.cc:344] Connected to a master server at 127.12.201.254:37865
I20250903 08:08:18.284636 18281 heartbeater.cc:461] Registering TS with master...
I20250903 08:08:18.285059 18281 heartbeater.cc:507] Master 127.12.201.254:37865 requested a full tablet report, sending...
I20250903 08:08:18.285938 17819 ts_manager.cc:194] Registered new tserver with Master: f6ad3857e5914ab8aca359575d246d4a (127.12.201.195:32805)
I20250903 08:08:18.286446 17819 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.201.195@KRBTEST.COM'} at 127.12.201.195:45683
I20250903 08:08:18.291314 13095 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250903 08:08:18.299566 17858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:18.293205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40238 (local address 127.12.201.254:37865)
0903 08:08:18.293353 (+ 148us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:18.293357 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:18.293518 (+ 161us) server_negotiation.cc:408] Connection header received
0903 08:08:18.293584 (+ 66us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:18.293586 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:18.293628 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:18.293706 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:18.294528 (+ 822us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:18.295106 (+ 578us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:18.295736 (+ 630us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:18.295871 (+ 135us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:18.296928 (+ 1057us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:18.296942 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:18.296945 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:08:18.296969 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:18.298208 (+ 1239us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:18.298570 (+ 362us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:18.298572 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:18.298574 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:18.298611 (+ 37us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:18.298777 (+ 166us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:18.298779 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:18.298780 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:18.298932 (+ 152us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:18.299102 (+ 170us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:18.299274 (+ 172us) server_negotiation.cc:300] Negotiation successful
0903 08:08:18.299378 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20250903 08:08:18.302073 17819 catalog_manager.cc:2232] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40238:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 08:08:18.304418 17819 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 08:08:18.316552 18292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:18.312142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.194:38235 (local address 127.0.0.1:49924)
0903 08:08:18.312500 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:18.312537 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:18.312666 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:18.313515 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:18.313523 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:18.313550 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 08:08:18.313842 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:18.313849 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:18.315124 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:18.315127 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:18.316047 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:18.316056 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:18.316168 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:18.316221 (+ 53us) client_negotiation.cc:769] Sending connection context
0903 08:08:18.316309 (+ 88us) client_negotiation.cc:240] Negotiation successful
0903 08:08:18.316396 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":113,"threads_started":1}
I20250903 08:08:18.317255 18295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:18.312850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.195:32805 (local address 127.0.0.1:54146)
0903 08:08:18.313487 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:18.313500 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:18.313561 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:18.314488 (+ 927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:18.314491 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:18.314511 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 08:08:18.314753 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:18.314759 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:18.315822 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:18.315825 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:18.316637 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:18.316648 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:18.316922 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:18.316951 (+ 29us) client_negotiation.cc:769] Sending connection context
0903 08:08:18.317003 (+ 52us) client_negotiation.cc:240] Negotiation successful
0903 08:08:18.317117 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":92,"threads_started":1}
I20250903 08:08:18.317428 18296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:18.313047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47046 (local address 127.12.201.193:46185)
0903 08:08:18.313354 (+ 307us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:18.313360 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:18.313382 (+ 22us) server_negotiation.cc:408] Connection header received
0903 08:08:18.313463 (+ 81us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:18.313467 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:18.313622 (+ 155us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:18.313774 (+ 152us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 08:08:18.314098 (+ 324us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:18.315105 (+ 1007us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:18.316481 (+ 1376us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:18.317027 (+ 546us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:18.317139 (+ 112us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:18.317201 (+ 62us) server_negotiation.cc:300] Negotiation successful
0903 08:08:18.317287 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":91,"threads_started":1}
I20250903 08:08:18.317746 18293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:18.312350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.193:46185 (local address 127.0.0.1:47046)
0903 08:08:18.312677 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:18.312691 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:18.312781 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:18.313756 (+ 975us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:18.313759 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:18.313774 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 08:08:18.313972 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:18.313979 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:18.315223 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:18.315226 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:18.316057 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:18.316064 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:18.316167 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:18.316221 (+ 54us) client_negotiation.cc:769] Sending connection context
0903 08:08:18.316309 (+ 88us) client_negotiation.cc:240] Negotiation successful
0903 08:08:18.316396 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":102,"threads_started":1}
I20250903 08:08:18.317871 18294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:18.312329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49924 (local address 127.12.201.194:38235)
0903 08:08:18.313027 (+ 698us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:18.313034 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:18.313057 (+ 23us) server_negotiation.cc:408] Connection header received
0903 08:08:18.313145 (+ 88us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:18.313153 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:18.313340 (+ 187us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:18.313987 (+ 647us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 08:08:18.314121 (+ 134us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:18.315024 (+ 903us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:18.316916 (+ 1892us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:18.317456 (+ 540us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:18.317567 (+ 111us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:18.317629 (+ 62us) server_negotiation.cc:300] Negotiation successful
0903 08:08:18.317724 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":131,"threads_started":1}
I20250903 08:08:18.318087 18297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:18.313177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54146 (local address 127.12.201.195:32805)
0903 08:08:18.314102 (+ 925us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:18.314108 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:18.314130 (+ 22us) server_negotiation.cc:408] Connection header received
0903 08:08:18.314206 (+ 76us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:18.314212 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:18.314352 (+ 140us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:18.314476 (+ 124us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 08:08:18.314869 (+ 393us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:18.315695 (+ 826us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:18.316940 (+ 1245us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:18.317633 (+ 693us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:18.317776 (+ 143us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:18.317844 (+ 68us) server_negotiation.cc:300] Negotiation successful
0903 08:08:18.317927 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":821,"thread_start_us":123,"threads_started":1}
I20250903 08:08:18.320633 17945 tablet_service.cc:1468] Processing CreateTablet for tablet 737dbb2ee6c24654a0e488f87f53071c (DEFAULT_TABLE table=test-table [id=72699a7a48c94d6f8eff0c0ba66fbd6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:08:18.320626 18215 tablet_service.cc:1468] Processing CreateTablet for tablet 737dbb2ee6c24654a0e488f87f53071c (DEFAULT_TABLE table=test-table [id=72699a7a48c94d6f8eff0c0ba66fbd6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:08:18.320724 18080 tablet_service.cc:1468] Processing CreateTablet for tablet 737dbb2ee6c24654a0e488f87f53071c (DEFAULT_TABLE table=test-table [id=72699a7a48c94d6f8eff0c0ba66fbd6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:08:18.321487 17945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 737dbb2ee6c24654a0e488f87f53071c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:08:18.321506 18080 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 737dbb2ee6c24654a0e488f87f53071c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:08:18.321506 18215 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 737dbb2ee6c24654a0e488f87f53071c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:08:18.326740 18298 tablet_bootstrap.cc:492] T 737dbb2ee6c24654a0e488f87f53071c P 51b1391323444680bed816270d464dc2: Bootstrap starting.
I20250903 08:08:18.328007 18299 tablet_bootstrap.cc:492] T 737dbb2ee6c24654a0e488f87f53071c P 76674bec22a54f67aa16f1e68af86408: Bootstrap starting.
I20250903 08:08:18.328171 18300 tablet_bootstrap.cc:492] T 737dbb2ee6c24654a0e488f87f53071c P f6ad3857e5914ab8aca359575d246d4a: Bootstrap starting.
I20250903 08:08:18.329015 18298 tablet_bootstrap.cc:654] T 737dbb2ee6c24654a0e488f87f53071c P 51b1391323444680bed816270d464dc2: Neither blocks nor log segments found. Creating new log.
I20250903 08:08:18.329735 18298 log.cc:826] T 737dbb2ee6c24654a0e488f87f53071c P 51b1391323444680bed816270d464dc2: Log is configured to *not* fsync() on all Append() calls
I20250903 08:08:18.330050 18299 tablet_bootstrap.cc:654] T 737dbb2ee6c24654a0e488f87f53071c P 76674bec22a54f67aa16f1e68af86408: Neither blocks nor log segments found. Creating new log.
I20250903 08:08:18.330050 18300 tablet_bootstrap.cc:654] T 737dbb2ee6c24654a0e488f87f53071c P f6ad3857e5914ab8aca359575d246d4a: Neither blocks nor log segments found. Creating new log.
I20250903 08:08:18.330834 18299 log.cc:826] T 737dbb2ee6c24654a0e488f87f53071c P 76674bec22a54f67aa16f1e68af86408: Log is configured to *not* fsync() on all Append() calls
I20250903 08:08:18.330834 18300 log.cc:826] T 737dbb2ee6c24654a0e488f87f53071c P f6ad3857e5914ab8aca359575d246d4a: Log is configured to *not* fsync() on all Append() calls
I20250903 08:08:18.332554 18300 tablet_bootstrap.cc:492] T 737dbb2ee6c24654a0e488f87f53071c P f6ad3857e5914ab8aca359575d246d4a: No bootstrap required, opened a new log
I20250903 08:08:18.332554 18299 tablet_bootstrap.cc:492] T 737dbb2ee6c24654a0e488f87f53071c P 76674bec22a54f67aa16f1e68af86408: No bootstrap required, opened a new log
I20250903 08:08:18.332734 18299 ts_tablet_manager.cc:1397] T 737dbb2ee6c24654a0e488f87f53071c P 76674bec22a54f67aa16f1e68af86408: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20250903 08:08:18.332743 18300 ts_tablet_manager.cc:1397] T 737dbb2ee6c24654a0e488f87f53071c P f6ad3857e5914ab8aca359575d246d4a: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20250903 08:08:18.332877 18298 tablet_bootstrap.cc:492] T 737dbb2ee6c24654a0e488f87f53071c P 51b1391323444680bed816270d464dc2: No bootstrap required, opened a new log
I20250903 08:08:18.333127 18298 ts_tablet_manager.cc:1397] T 737dbb2ee6c24654a0e488f87f53071c P 51b1391323444680bed816270d464dc2: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20250903 08:08:18.336042 18298 raft_consensus.cc:357] T 737dbb2ee6c24654a0e488f87f53071c P 51b1391323444680bed816270d464dc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76674bec22a54f67aa16f1e68af86408" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 38235 } } peers { permanent_uuid: "f6ad3857e5914ab8aca359575d246d4a" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 32805 } } peers { permanent_uuid: "51b1391323444680bed816270d464dc2" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 46185 } }
I20250903 08:08:18.336230 18298 raft_consensus.cc:383] T 737dbb2ee6c24654a0e488f87f53071c P 51b1391323444680bed816270d464dc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:08:18.336290 18298 raft_consensus.cc:738] T 737dbb2ee6c24654a0e488f87f53071c P 51b1391323444680bed816270d464dc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51b1391323444680bed816270d464dc2, State: Initialized, Role: FOLLOWER
I20250903 08:08:18.336329 18300 raft_consensus.cc:357] T 737dbb2ee6c24654a0e488f87f53071c P f6ad3857e5914ab8aca359575d246d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76674bec22a54f67aa16f1e68af86408" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 38235 } } peers { permanent_uuid: "f6ad3857e5914ab8aca359575d246d4a" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 32805 } } peers { permanent_uuid: "51b1391323444680bed816270d464dc2" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 46185 } }
I20250903 08:08:18.336328 18299 raft_consensus.cc:357] T 737dbb2ee6c24654a0e488f87f53071c P 76674bec22a54f67aa16f1e68af86408 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76674bec22a54f67aa16f1e68af86408" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 38235 } } peers { permanent_uuid: "f6ad3857e5914ab8aca359575d246d4a" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 32805 } } peers { permanent_uuid: "51b1391323444680bed816270d464dc2" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 46185 } }
I20250903 08:08:18.336524 18299 raft_consensus.cc:383] T 737dbb2ee6c24654a0e488f87f53071c P 76674bec22a54f67aa16f1e68af86408 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:08:18.336524 18300 raft_consensus.cc:383] T 737dbb2ee6c24654a0e488f87f53071c P f6ad3857e5914ab8aca359575d246d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:08:18.336587 18299 raft_consensus.cc:738] T 737dbb2ee6c24654a0e488f87f53071c P 76674bec22a54f67aa16f1e68af86408 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76674bec22a54f67aa16f1e68af86408, State: Initialized, Role: FOLLOWER
I20250903 08:08:18.336588 18300 raft_consensus.cc:738] T 737dbb2ee6c24654a0e488f87f53071c P f6ad3857e5914ab8aca359575d246d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6ad3857e5914ab8aca359575d246d4a, State: Initialized, Role: FOLLOWER
I20250903 08:08:18.336716 18298 consensus_queue.cc:260] T 737dbb2ee6c24654a0e488f87f53071c P 51b1391323444680bed816270d464dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76674bec22a54f67aa16f1e68af86408" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 38235 } } peers { permanent_uuid: "f6ad3857e5914ab8aca359575d246d4a" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 32805 } } peers { permanent_uuid: "51b1391323444680bed816270d464dc2" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 46185 } }
I20250903 08:08:18.337008 18300 consensus_queue.cc:260] T 737dbb2ee6c24654a0e488f87f53071c P f6ad3857e5914ab8aca359575d246d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76674bec22a54f67aa16f1e68af86408" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 38235 } } peers { permanent_uuid: "f6ad3857e5914ab8aca359575d246d4a" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 32805 } } peers { permanent_uuid: "51b1391323444680bed816270d464dc2" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 46185 } }
I20250903 08:08:18.337008 18299 consensus_queue.cc:260] T 737dbb2ee6c24654a0e488f87f53071c P 76674bec22a54f67aa16f1e68af86408 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76674bec22a54f67aa16f1e68af86408" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 38235 } } peers { permanent_uuid: "f6ad3857e5914ab8aca359575d246d4a" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 32805 } } peers { permanent_uuid: "51b1391323444680bed816270d464dc2" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 46185 } }
I20250903 08:08:18.337380 18011 heartbeater.cc:499] Master 127.12.201.254:37865 was elected leader, sending a full tablet report...
I20250903 08:08:18.337874 18300 ts_tablet_manager.cc:1428] T 737dbb2ee6c24654a0e488f87f53071c P f6ad3857e5914ab8aca359575d246d4a: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20250903 08:08:18.337874 18299 ts_tablet_manager.cc:1428] T 737dbb2ee6c24654a0e488f87f53071c P 76674bec22a54f67aa16f1e68af86408: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20250903 08:08:18.338235 18281 heartbeater.cc:499] Master 127.12.201.254:37865 was elected leader, sending a full tablet report...
I20250903 08:08:18.338403 18146 heartbeater.cc:499] Master 127.12.201.254:37865 was elected leader, sending a full tablet report...
I20250903 08:08:18.338783 18298 ts_tablet_manager.cc:1428] T 737dbb2ee6c24654a0e488f87f53071c P 51b1391323444680bed816270d464dc2: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250903 08:08:18.384227 18306 raft_consensus.cc:491] T 737dbb2ee6c24654a0e488f87f53071c P 76674bec22a54f67aa16f1e68af86408 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 08:08:18.384419 18306 raft_consensus.cc:513] T 737dbb2ee6c24654a0e488f87f53071c P 76674bec22a54f67aa16f1e68af86408 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76674bec22a54f67aa16f1e68af86408" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 38235 } } peers { permanent_uuid: "f6ad3857e5914ab8aca359575d246d4a" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 32805 } } peers { permanent_uuid: "51b1391323444680bed816270d464dc2" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 46185 } }
I20250903 08:08:18.385524 18306 leader_election.cc:290] T 737dbb2ee6c24654a0e488f87f53071c P 76674bec22a54f67aa16f1e68af86408 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6ad3857e5914ab8aca359575d246d4a (127.12.201.195:32805), 51b1391323444680bed816270d464dc2 (127.12.201.193:46185)
I20250903 08:08:18.389174 18307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:18.385790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.193:46185 (local address 127.12.201.194:34593)
0903 08:08:18.386165 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:18.386178 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:18.386255 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:18.386500 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:18.386503 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:18.386526 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 08:08:18.386786 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:18.386792 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:18.387731 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:18.387734 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:18.388673 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:18.388685 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:18.388815 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:18.388838 (+ 23us) client_negotiation.cc:769] Sending connection context
0903 08:08:18.388897 (+ 59us) client_negotiation.cc:240] Negotiation successful
0903 08:08:18.388948 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":127,"threads_started":1}
I20250903 08:08:18.389186 18148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:18.385758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.195:32805 (local address 127.12.201.194:58191)
0903 08:08:18.385977 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:18.385993 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:18.386097 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:18.386379 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:18.386381 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:18.386394 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 08:08:18.386652 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:18.386661 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:18.387679 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:18.387683 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:18.388695 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:18.388701 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:18.388815 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:18.388838 (+ 23us) client_negotiation.cc:769] Sending connection context
0903 08:08:18.388889 (+ 51us) client_negotiation.cc:240] Negotiation successful
0903 08:08:18.388948 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149}
I20250903 08:08:18.389555 18296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:18.385870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.194:34593 (local address 127.12.201.193:46185)
0903 08:08:18.386005 (+ 135us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:18.386010 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:18.386248 (+ 238us) server_negotiation.cc:408] Connection header received
0903 08:08:18.386332 (+ 84us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:18.386338 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:18.386393 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:18.386469 (+ 76us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 08:08:18.386890 (+ 421us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:18.387617 (+ 727us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:18.388808 (+ 1191us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:18.389271 (+ 463us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:18.389314 (+ 43us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:18.389368 (+ 54us) server_negotiation.cc:300] Negotiation successful
0903 08:08:18.389449 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20250903 08:08:18.389837 18297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:18.385850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.194:58191 (local address 127.12.201.195:32805)
0903 08:08:18.385976 (+ 126us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:18.385981 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:18.386071 (+ 90us) server_negotiation.cc:408] Connection header received
0903 08:08:18.386203 (+ 132us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:18.386206 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:18.386264 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:18.386352 (+ 88us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 08:08:18.386770 (+ 418us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:18.387559 (+ 789us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:18.389029 (+ 1470us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:18.389559 (+ 530us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:18.389590 (+ 31us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:18.389642 (+ 52us) server_negotiation.cc:300] Negotiation successful
0903 08:08:18.389712 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20250903 08:08:18.390344 17965 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "737dbb2ee6c24654a0e488f87f53071c" candidate_uuid: "76674bec22a54f67aa16f1e68af86408" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51b1391323444680bed816270d464dc2" is_pre_election: true
I20250903 08:08:18.390381 18235 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "737dbb2ee6c24654a0e488f87f53071c" candidate_uuid: "76674bec22a54f67aa16f1e68af86408" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6ad3857e5914ab8aca359575d246d4a" is_pre_election: true
I20250903 08:08:18.390616 18235 raft_consensus.cc:2466] T 737dbb2ee6c24654a0e488f87f53071c P f6ad3857e5914ab8aca359575d246d4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76674bec22a54f67aa16f1e68af86408 in term 0.
I20250903 08:08:18.390623 17965 raft_consensus.cc:2466] T 737dbb2ee6c24654a0e488f87f53071c P 51b1391323444680bed816270d464dc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76674bec22a54f67aa16f1e68af86408 in term 0.
I20250903 08:08:18.391109 18033 leader_election.cc:304] T 737dbb2ee6c24654a0e488f87f53071c P 76674bec22a54f67aa16f1e68af86408 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51b1391323444680bed816270d464dc2, 76674bec22a54f67aa16f1e68af86408; no voters:
I20250903 08:08:18.391348 18306 raft_consensus.cc:2802] T 737dbb2ee6c24654a0e488f87f53071c P 76674bec22a54f67aa16f1e68af86408 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 08:08:18.391418 18306 raft_consensus.cc:491] T 737dbb2ee6c24654a0e488f87f53071c P 76674bec22a54f67aa16f1e68af86408 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 08:08:18.391449 18306 raft_consensus.cc:3058] T 737dbb2ee6c24654a0e488f87f53071c P 76674bec22a54f67aa16f1e68af86408 [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:08:18.392405 18306 raft_consensus.cc:513] T 737dbb2ee6c24654a0e488f87f53071c P 76674bec22a54f67aa16f1e68af86408 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76674bec22a54f67aa16f1e68af86408" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 38235 } } peers { permanent_uuid: "f6ad3857e5914ab8aca359575d246d4a" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 32805 } } peers { permanent_uuid: "51b1391323444680bed816270d464dc2" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 46185 } }
I20250903 08:08:18.392800 18306 leader_election.cc:290] T 737dbb2ee6c24654a0e488f87f53071c P 76674bec22a54f67aa16f1e68af86408 [CANDIDATE]: Term 1 election: Requested vote from peers f6ad3857e5914ab8aca359575d246d4a (127.12.201.195:32805), 51b1391323444680bed816270d464dc2 (127.12.201.193:46185)
I20250903 08:08:18.393095 18235 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "737dbb2ee6c24654a0e488f87f53071c" candidate_uuid: "76674bec22a54f67aa16f1e68af86408" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6ad3857e5914ab8aca359575d246d4a"
I20250903 08:08:18.393183 17965 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "737dbb2ee6c24654a0e488f87f53071c" candidate_uuid: "76674bec22a54f67aa16f1e68af86408" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51b1391323444680bed816270d464dc2"
I20250903 08:08:18.393203 18235 raft_consensus.cc:3058] T 737dbb2ee6c24654a0e488f87f53071c P f6ad3857e5914ab8aca359575d246d4a [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:08:18.393301 17965 raft_consensus.cc:3058] T 737dbb2ee6c24654a0e488f87f53071c P 51b1391323444680bed816270d464dc2 [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:08:18.394169 18235 raft_consensus.cc:2466] T 737dbb2ee6c24654a0e488f87f53071c P f6ad3857e5914ab8aca359575d246d4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76674bec22a54f67aa16f1e68af86408 in term 1.
I20250903 08:08:18.394261 17965 raft_consensus.cc:2466] T 737dbb2ee6c24654a0e488f87f53071c P 51b1391323444680bed816270d464dc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76674bec22a54f67aa16f1e68af86408 in term 1.
I20250903 08:08:18.394498 18034 leader_election.cc:304] T 737dbb2ee6c24654a0e488f87f53071c P 76674bec22a54f67aa16f1e68af86408 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76674bec22a54f67aa16f1e68af86408, f6ad3857e5914ab8aca359575d246d4a; no voters:
I20250903 08:08:18.394662 18306 raft_consensus.cc:2802] T 737dbb2ee6c24654a0e488f87f53071c P 76674bec22a54f67aa16f1e68af86408 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 08:08:18.394843 18306 raft_consensus.cc:695] T 737dbb2ee6c24654a0e488f87f53071c P 76674bec22a54f67aa16f1e68af86408 [term 1 LEADER]: Becoming Leader. State: Replica: 76674bec22a54f67aa16f1e68af86408, State: Running, Role: LEADER
I20250903 08:08:18.395169 18306 consensus_queue.cc:237] T 737dbb2ee6c24654a0e488f87f53071c P 76674bec22a54f67aa16f1e68af86408 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76674bec22a54f67aa16f1e68af86408" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 38235 } } peers { permanent_uuid: "f6ad3857e5914ab8aca359575d246d4a" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 32805 } } peers { permanent_uuid: "51b1391323444680bed816270d464dc2" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 46185 } }
I20250903 08:08:18.398118 17818 catalog_manager.cc:5582] T 737dbb2ee6c24654a0e488f87f53071c P 76674bec22a54f67aa16f1e68af86408 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76674bec22a54f67aa16f1e68af86408 (127.12.201.194). New cstate: current_term: 1 leader_uuid: "76674bec22a54f67aa16f1e68af86408" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76674bec22a54f67aa16f1e68af86408" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 38235 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6ad3857e5914ab8aca359575d246d4a" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 32805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51b1391323444680bed816270d464dc2" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 46185 } health_report { overall_health: UNKNOWN } } }
I20250903 08:08:18.414801 18294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:18.411986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49936 (local address 127.12.201.194:38235)
0903 08:08:18.412096 (+ 110us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:18.412100 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:18.412113 (+ 13us) server_negotiation.cc:408] Connection header received
0903 08:08:18.412240 (+ 127us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:18.412243 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:18.412300 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:18.412378 (+ 78us) server_negotiation.cc:227] Negotiated authn=TOKEN
0903 08:08:18.412648 (+ 270us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:18.413165 (+ 517us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:18.413862 (+ 697us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:18.413994 (+ 132us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:18.414077 (+ 83us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0903 08:08:18.414486 (+ 409us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0903 08:08:18.414576 (+ 90us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:18.414650 (+ 74us) server_negotiation.cc:300] Negotiation successful
0903 08:08:18.414692 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20250903 08:08:18.420059 18235 raft_consensus.cc:1273] T 737dbb2ee6c24654a0e488f87f53071c P f6ad3857e5914ab8aca359575d246d4a [term 1 FOLLOWER]: Refusing update from remote peer 76674bec22a54f67aa16f1e68af86408: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 08:08:18.420071 17965 raft_consensus.cc:1273] T 737dbb2ee6c24654a0e488f87f53071c P 51b1391323444680bed816270d464dc2 [term 1 FOLLOWER]: Refusing update from remote peer 76674bec22a54f67aa16f1e68af86408: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 08:08:18.420682 18308 consensus_queue.cc:1035] T 737dbb2ee6c24654a0e488f87f53071c P 76674bec22a54f67aa16f1e68af86408 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51b1391323444680bed816270d464dc2" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 08:08:18.420846 18306 consensus_queue.cc:1035] T 737dbb2ee6c24654a0e488f87f53071c P 76674bec22a54f67aa16f1e68af86408 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6ad3857e5914ab8aca359575d246d4a" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 32805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 08:08:18.427081 18311 mvcc.cc:204] Tried to move back new op lower bound from 7196208735921045504 to 7196208735828336640. Current Snapshot: MvccSnapshot[applied={T|T < 7196208735921045504}]
I20250903 08:08:18.428903 18313 mvcc.cc:204] Tried to move back new op lower bound from 7196208735921045504 to 7196208735828336640. Current Snapshot: MvccSnapshot[applied={T|T < 7196208735921045504}]
I20250903 08:08:18.429059 18312 mvcc.cc:204] Tried to move back new op lower bound from 7196208735921045504 to 7196208735828336640. Current Snapshot: MvccSnapshot[applied={T|T < 7196208735921045504}]
I20250903 08:08:18.435141 17818 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40238:
table { table_name: "test-table" } modify_external_catalogs: true
I20250903 08:08:18.435317 17818 catalog_manager.cc:2730] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40238:
table { table_name: "test-table" } modify_external_catalogs: true
I20250903 08:08:18.437594 17818 catalog_manager.cc:5869] T 00000000000000000000000000000000 P b18a2a441e5b49a0ac165065b5fbf71b: Sending DeleteTablet for 3 replicas of tablet 737dbb2ee6c24654a0e488f87f53071c
I20250903 08:08:18.438309 17945 tablet_service.cc:1515] Processing DeleteTablet for tablet 737dbb2ee6c24654a0e488f87f53071c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 08:08:18 UTC) from {username='kudu', principal='kudu/127.12.201.254@KRBTEST.COM'} at 127.0.0.1:47046
I20250903 08:08:18.438308 18215 tablet_service.cc:1515] Processing DeleteTablet for tablet 737dbb2ee6c24654a0e488f87f53071c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 08:08:18 UTC) from {username='kudu', principal='kudu/127.12.201.254@KRBTEST.COM'} at 127.0.0.1:54146
I20250903 08:08:18.438771 18320 tablet_replica.cc:331] T 737dbb2ee6c24654a0e488f87f53071c P 51b1391323444680bed816270d464dc2: stopping tablet replica
I20250903 08:08:18.438747 18080 tablet_service.cc:1515] Processing DeleteTablet for tablet 737dbb2ee6c24654a0e488f87f53071c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 08:08:18 UTC) from {username='kudu', principal='kudu/127.12.201.254@KRBTEST.COM'} at 127.0.0.1:49924
I20250903 08:08:18.439064 18320 raft_consensus.cc:2241] T 737dbb2ee6c24654a0e488f87f53071c P 51b1391323444680bed816270d464dc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 08:08:18.439216 18322 tablet_replica.cc:331] T 737dbb2ee6c24654a0e488f87f53071c P 76674bec22a54f67aa16f1e68af86408: stopping tablet replica
I20250903 08:08:18.439378 18320 raft_consensus.cc:2270] T 737dbb2ee6c24654a0e488f87f53071c P 51b1391323444680bed816270d464dc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 08:08:18.439531 18322 raft_consensus.cc:2241] T 737dbb2ee6c24654a0e488f87f53071c P 76674bec22a54f67aa16f1e68af86408 [term 1 LEADER]: Raft consensus shutting down.
I20250903 08:08:18.439705 18321 tablet_replica.cc:331] T 737dbb2ee6c24654a0e488f87f53071c P f6ad3857e5914ab8aca359575d246d4a: stopping tablet replica
I20250903 08:08:18.439769 13095 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUML90r/build/debug/bin/kudu with pid 17883
I20250903 08:08:18.439960 18321 raft_consensus.cc:2241] T 737dbb2ee6c24654a0e488f87f53071c P f6ad3857e5914ab8aca359575d246d4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 08:08:18.440335 18321 raft_consensus.cc:2270] T 737dbb2ee6c24654a0e488f87f53071c P f6ad3857e5914ab8aca359575d246d4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 08:08:18.441545 18322 raft_consensus.cc:2270] T 737dbb2ee6c24654a0e488f87f53071c P 76674bec22a54f67aa16f1e68af86408 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 08:08:18.442507 18321 ts_tablet_manager.cc:1905] T 737dbb2ee6c24654a0e488f87f53071c P f6ad3857e5914ab8aca359575d246d4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 08:08:18.442884 18322 ts_tablet_manager.cc:1905] T 737dbb2ee6c24654a0e488f87f53071c P 76674bec22a54f67aa16f1e68af86408: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 08:08:18.445411 18322 ts_tablet_manager.cc:1918] T 737dbb2ee6c24654a0e488f87f53071c P 76674bec22a54f67aa16f1e68af86408: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250903 08:08:18.445562 18322 log.cc:1199] T 737dbb2ee6c24654a0e488f87f53071c P 76674bec22a54f67aa16f1e68af86408: Deleting WAL directory at /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/minicluster-data/ts-1/wal/wals/737dbb2ee6c24654a0e488f87f53071c
I20250903 08:08:18.445636 18321 ts_tablet_manager.cc:1918] T 737dbb2ee6c24654a0e488f87f53071c P f6ad3857e5914ab8aca359575d246d4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250903 08:08:18.445727 18321 log.cc:1199] T 737dbb2ee6c24654a0e488f87f53071c P f6ad3857e5914ab8aca359575d246d4a: Deleting WAL directory at /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756886860253814-13095-0/minicluster-data/ts-2/wal/wals/737dbb2ee6c24654a0e488f87f53071c
I20250903 08:08:18.446055 18321 ts_tablet_manager.cc:1939] T 737dbb2ee6c24654a0e488f87f53071c P f6ad3857e5914ab8aca359575d246d4a: Deleting consensus metadata
I20250903 08:08:18.446148 18322 ts_tablet_manager.cc:1939] T 737dbb2ee6c24654a0e488f87f53071c P 76674bec22a54f67aa16f1e68af86408: Deleting consensus metadata
I20250903 08:08:18.447194 17804 catalog_manager.cc:4928] TS f6ad3857e5914ab8aca359575d246d4a (127.12.201.195:32805): tablet 737dbb2ee6c24654a0e488f87f53071c (table test-table [id=72699a7a48c94d6f8eff0c0ba66fbd6e]) successfully deleted
I20250903 08:08:18.447372 17804 catalog_manager.cc:4928] TS 76674bec22a54f67aa16f1e68af86408 (127.12.201.194:38235): tablet 737dbb2ee6c24654a0e488f87f53071c (table test-table [id=72699a7a48c94d6f8eff0c0ba66fbd6e]) successfully deleted
W20250903 08:08:18.447463 17803 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.201.193:46185 (error 108)
I20250903 08:08:18.447629 13095 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUML90r/build/debug/bin/kudu with pid 18018
I20250903 08:08:18.449779 18293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:18.449523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.193:46185 (local address 127.0.0.1:60078)
0903 08:08:18.449649 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:18.449703 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.201.193:46185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20250903 08:08:18.449903 17803 catalog_manager.cc:4671] TS 51b1391323444680bed816270d464dc2 (127.12.201.193:46185): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 737dbb2ee6c24654a0e488f87f53071c: Network error: Client connection negotiation failed: client connection to 127.12.201.193:46185: connect: Connection refused (error 111)
I20250903 08:08:18.453117 13095 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUML90r/build/debug/bin/kudu with pid 18153
I20250903 08:08:18.458694 13095 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUML90r/build/debug/bin/kudu with pid 17788
2025-09-03T08:08:18Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (3222 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2025-09-03T08:08:18Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-03T08:08:18Z Disabled control of system clock
I20250903 08:08:18.487036 13095 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUML90r/build/debug/bin/kudu
/tmp/dist-test-taskUML90r/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756886860253814-13095-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756886860253814-13095-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756886860253814-13095-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756886860253814-13095-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.201.254:43709
--webserver_interface=127.12.201.254
--webserver_port=0
--builtin_ntp_servers=127.12.201.212:42435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.201.254:43709
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20250903 08:08:18.589629 18327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:08:18.589883 18327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:08:18.589931 18327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:08:18.592995 18327 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250903 08:08:18.593047 18327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 08:08:18.593063 18327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:08:18.593081 18327 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250903 08:08:18.593098 18327 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250903 08:08:18.596689 18327 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.201.212:42435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756886860253814-13095-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756886860253814-13095-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.201.254:43709
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.201.254:43709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756886860253814-13095-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.201.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756886860253814-13095-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 07:50:08 UTC on 5fd53c4cbb9d
build id 7874
I20250903 08:08:18.597647 18327 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:08:18.598726 18327 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:08:18.604049 18332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:18.604049 18335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:18.604053 18333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:08:18.604405 18327 server_base.cc:1047] running on GCE node
I20250903 08:08:18.604871 18327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:08:18.605712 18327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:08:18.606892 18327 hybrid_clock.cc:648] HybridClock initialized: now 1756886898606882 us; error 29 us; skew 500 ppm
I20250903 08:08:18.608647 18327 webserver.cc:480] Webserver started at http://127.12.201.254:34011/ using document root <none> and password file <none>
I20250903 08:08:18.609196 18327 fs_manager.cc:362] Metadata directory not provided
I20250903 08:08:18.609256 18327 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:08:18.609431 18327 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:08:18.611151 18327 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756886860253814-13095-0/minicluster-data/master-0/data/instance:
uuid: "c1056cbdf1a84732889259c48109686d"
format_stamp: "Formatted at 2025-09-03 08:08:18 on dist-test-slave-27r9"
server_key: "e3e861e9909023ee86346d27c1adc816"
server_key_iv: "9017dc2030483ec039bccf7610175b7c"
server_key_version: "encryptionkey@0"
I20250903 08:08:18.611610 18327 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756886860253814-13095-0/minicluster-data/master-0/wal/instance:
uuid: "c1056cbdf1a84732889259c48109686d"
format_stamp: "Formatted at 2025-09-03 08:08:18 on dist-test-slave-27r9"
server_key: "e3e861e9909023ee86346d27c1adc816"
server_key_iv: "9017dc2030483ec039bccf7610175b7c"
server_key_version: "encryptionkey@0"
I20250903 08:08:18.615041 18327 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250903 08:08:18.617200 18341 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:18.618229 18327 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250903 08:08:18.618340 18327 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756886860253814-13095-0/minicluster-data/master-0/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756886860253814-13095-0/minicluster-data/master-0/wal
uuid: "c1056cbdf1a84732889259c48109686d"
format_stamp: "Formatted at 2025-09-03 08:08:18 on dist-test-slave-27r9"
server_key: "e3e861e9909023ee86346d27c1adc816"
server_key_iv: "9017dc2030483ec039bccf7610175b7c"
server_key_version: "encryptionkey@0"
I20250903 08:08:18.618440 18327 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756886860253814-13095-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756886860253814-13095-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756886860253814-13095-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:08:18.623135 18327 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:08:18.623684 18327 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:08:18.623860 18327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:08:18.630795 18327 rpc_server.cc:307] RPC server started. Bound to: 127.12.201.254:43709
I20250903 08:08:18.630846 18393 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.201.254:43709 every 8 connection(s)
I20250903 08:08:18.631825 18327 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756886860253814-13095-0/minicluster-data/master-0/data/info.pb
I20250903 08:08:18.634223 18394 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:08:18.639178 18394 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1056cbdf1a84732889259c48109686d: Bootstrap starting.
I20250903 08:08:18.641274 18394 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1056cbdf1a84732889259c48109686d: Neither blocks nor log segments found. Creating new log.
I20250903 08:08:18.641649 13095 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUML90r/build/debug/bin/kudu as pid 18327
I20250903 08:08:18.641752 13095 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756886860253814-13095-0/minicluster-data/master-0/wal/instance
I20250903 08:08:18.641952 13095 external_mini_cluster.cc:1468] Setting key c9c24bc3baba09c4ac1e470deb87e23c
I20250903 08:08:18.641961 18394 log.cc:826] T 00000000000000000000000000000000 P c1056cbdf1a84732889259c48109686d: Log is configured to *not* fsync() on all Append() calls
I20250903 08:08:18.643913 18394 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1056cbdf1a84732889259c48109686d: No bootstrap required, opened a new log
I20250903 08:08:18.646723 18398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:18.642978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51062 (local address 127.12.201.254:43709)
0903 08:08:18.643573 (+ 595us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:18.643582 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:18.643612 (+ 30us) server_negotiation.cc:408] Connection header received
0903 08:08:18.644145 (+ 533us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:18.644163 (+ 18us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:18.644400 (+ 237us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:18.644646 (+ 246us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:18.645664 (+ 1018us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:18.645669 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:18.645678 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0903 08:08:18.645688 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:18.645726 (+ 38us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0903 08:08:18.645737 (+ 11us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:18.645845 (+ 108us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:18.645971 (+ 126us) server_negotiation.cc:300] Negotiation successful
0903 08:08:18.646196 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":127,"threads_started":1}
I20250903 08:08:18.646644 18394 raft_consensus.cc:357] T 00000000000000000000000000000000 P c1056cbdf1a84732889259c48109686d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1056cbdf1a84732889259c48109686d" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 43709 } }
I20250903 08:08:18.646821 18394 raft_consensus.cc:383] T 00000000000000000000000000000000 P c1056cbdf1a84732889259c48109686d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:08:18.646885 18394 raft_consensus.cc:738] T 00000000000000000000000000000000 P c1056cbdf1a84732889259c48109686d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1056cbdf1a84732889259c48109686d, State: Initialized, Role: FOLLOWER
I20250903 08:08:18.647322 18394 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1056cbdf1a84732889259c48109686d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1056cbdf1a84732889259c48109686d" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 43709 } }
I20250903 08:08:18.647457 18394 raft_consensus.cc:397] T 00000000000000000000000000000000 P c1056cbdf1a84732889259c48109686d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250903 08:08:18.647519 18394 raft_consensus.cc:491] T 00000000000000000000000000000000 P c1056cbdf1a84732889259c48109686d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250903 08:08:18.647639 18394 raft_consensus.cc:3058] T 00000000000000000000000000000000 P c1056cbdf1a84732889259c48109686d [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:08:18.648479 18394 raft_consensus.cc:513] T 00000000000000000000000000000000 P c1056cbdf1a84732889259c48109686d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1056cbdf1a84732889259c48109686d" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 43709 } }
I20250903 08:08:18.648792 18394 leader_election.cc:304] T 00000000000000000000000000000000 P c1056cbdf1a84732889259c48109686d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c1056cbdf1a84732889259c48109686d; no voters:
I20250903 08:08:18.649062 18394 leader_election.cc:290] T 00000000000000000000000000000000 P c1056cbdf1a84732889259c48109686d [CANDIDATE]: Term 1 election: Requested vote from peers
I20250903 08:08:18.649159 18399 raft_consensus.cc:2802] T 00000000000000000000000000000000 P c1056cbdf1a84732889259c48109686d [term 1 FOLLOWER]: Leader election won for term 1
I20250903 08:08:18.649358 18399 raft_consensus.cc:695] T 00000000000000000000000000000000 P c1056cbdf1a84732889259c48109686d [term 1 LEADER]: Becoming Leader. State: Replica: c1056cbdf1a84732889259c48109686d, State: Running, Role: LEADER
I20250903 08:08:18.649603 18399 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1056cbdf1a84732889259c48109686d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1056cbdf1a84732889259c48109686d" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 43709 } }
I20250903 08:08:18.650079 18394 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1056cbdf1a84732889259c48109686d [sys.catalog]: configured and running, proceeding with master startup.
I20250903 08:08:18.651005 18400 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1056cbdf1a84732889259c48109686d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c1056cbdf1a84732889259c48109686d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1056cbdf1a84732889259c48109686d" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 43709 } } }
I20250903 08:08:18.651137 18400 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1056cbdf1a84732889259c48109686d [sys.catalog]: This master's current role is: LEADER
I20250903 08:08:18.651630 18401 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1056cbdf1a84732889259c48109686d [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1056cbdf1a84732889259c48109686d. Latest consensus state: current_term: 1 leader_uuid: "c1056cbdf1a84732889259c48109686d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1056cbdf1a84732889259c48109686d" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 43709 } } }
I20250903 08:08:18.651731 18401 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1056cbdf1a84732889259c48109686d [sys.catalog]: This master's current role is: LEADER
I20250903 08:08:18.652212 18408 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250903 08:08:18.654593 18408 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250903 08:08:18.659603 18408 catalog_manager.cc:1349] Generated new cluster ID: a561376632a14dd5b291107388925f0d
I20250903 08:08:18.659677 18408 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250903 08:08:18.688895 18408 catalog_manager.cc:1372] Generated new certificate authority record
I20250903 08:08:18.689231 18408 catalog_manager.cc:1506] Loading token signing keys...
I20250903 08:08:18.696714 18408 catalog_manager.cc:5955] T 00000000000000000000000000000000 P c1056cbdf1a84732889259c48109686d: Generated new TSK 0
I20250903 08:08:18.697463 18408 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250903 08:08:18.700762 13095 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUML90r/build/debug/bin/kudu
/tmp/dist-test-taskUML90r/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756886860253814-13095-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756886860253814-13095-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756886860253814-13095-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756886860253814-13095-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.201.193:0
--local_ip_for_outbound_sockets=127.12.201.193
--webserver_interface=127.12.201.193
--webserver_port=0
--tserver_master_addrs=127.12.201.254:43709
--builtin_ntp_servers=127.12.201.212:42435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20250903 08:08:18.806111 18418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:08:18.806329 18418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:08:18.806386 18418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:08:18.809721 18418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 08:08:18.809779 18418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:08:18.809861 18418 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.201.193
I20250903 08:08:18.813596 18418 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.201.212:42435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756886860253814-13095-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756886860253814-13095-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.201.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756886860253814-13095-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.201.193
--webserver_port=0
--tserver_master_addrs=127.12.201.254:43709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18418
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.201.193
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756886860253814-13095-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 07:50:08 UTC on 5fd53c4cbb9d
build id 7874
I20250903 08:08:18.814685 18418 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:08:18.815843 18418 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:08:18.822118 18426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:18.822216 18424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:08:18.822315 18418 server_base.cc:1047] running on GCE node
W20250903 08:08:18.822328 18423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:08:18.822719 18418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:08:18.823249 18418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:08:18.824410 18418 hybrid_clock.cc:648] HybridClock initialized: now 1756886898824393 us; error 29 us; skew 500 ppm
I20250903 08:08:18.826082 18418 webserver.cc:480] Webserver started at http://127.12.201.193:38441/ using document root <none> and password file <none>
I20250903 08:08:18.826620 18418 fs_manager.cc:362] Metadata directory not provided
I20250903 08:08:18.826673 18418 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:08:18.826879 18418 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:08:18.828579 18418 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756886860253814-13095-0/minicluster-data/ts-0/data/instance:
uuid: "3e70e903dff24329baceddb7b39a43ba"
format_stamp: "Formatted at 2025-09-03 08:08:18 on dist-test-slave-27r9"
server_key: "87f45124eb745f6f0bd1ce386c3a8892"
server_key_iv: "4b1b75282dab1e30e9ee85d6b7d7afbd"
server_key_version: "encryptionkey@0"
I20250903 08:08:18.829061 18418 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756886860253814-13095-0/minicluster-data/ts-0/wal/instance:
uuid: "3e70e903dff24329baceddb7b39a43ba"
format_stamp: "Formatted at 2025-09-03 08:08:18 on dist-test-slave-27r9"
server_key: "87f45124eb745f6f0bd1ce386c3a8892"
server_key_iv: "4b1b75282dab1e30e9ee85d6b7d7afbd"
server_key_version: "encryptionkey@0"
I20250903 08:08:18.832309 18418 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250903 08:08:18.834412 18432 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:18.835438 18418 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250903 08:08:18.835547 18418 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756886860253814-13095-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756886860253814-13095-0/minicluster-data/ts-0/wal
uuid: "3e70e903dff24329baceddb7b39a43ba"
format_stamp: "Formatted at 2025-09-03 08:08:18 on dist-test-slave-27r9"
server_key: "87f45124eb745f6f0bd1ce386c3a8892"
server_key_iv: "4b1b75282dab1e30e9ee85d6b7d7afbd"
server_key_version: "encryptionkey@0"
I20250903 08:08:18.835633 18418 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756886860253814-13095-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756886860253814-13095-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756886860253814-13095-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:08:18.840198 18418 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:08:18.840721 18418 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:08:18.840893 18418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:08:18.841388 18418 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 08:08:18.842248 18418 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 08:08:18.842298 18418 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:18.842350 18418 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 08:08:18.842388 18418 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:18.851464 18418 rpc_server.cc:307] RPC server started. Bound to: 127.12.201.193:45535
I20250903 08:08:18.851498 18545 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.201.193:45535 every 8 connection(s)
I20250903 08:08:18.852425 18418 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756886860253814-13095-0/minicluster-data/ts-0/data/info.pb
I20250903 08:08:18.856386 13095 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUML90r/build/debug/bin/kudu as pid 18418
I20250903 08:08:18.856493 13095 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756886860253814-13095-0/minicluster-data/ts-0/wal/instance
I20250903 08:08:18.856796 13095 external_mini_cluster.cc:1468] Setting key adde7b0ec15e754521fbe4124610a2b8
I20250903 08:08:18.857816 18398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:18.854155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.193:47381 (local address 127.12.201.254:43709)
0903 08:08:18.854300 (+ 145us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:18.854304 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:18.855172 (+ 868us) server_negotiation.cc:408] Connection header received
0903 08:08:18.855964 (+ 792us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:18.855967 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:18.856017 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:18.856095 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:18.857322 (+ 1227us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:18.857326 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:18.857330 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0903 08:08:18.857348 (+ 18us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:18.857373 (+ 25us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0903 08:08:18.857383 (+ 10us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:18.857482 (+ 99us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:18.857669 (+ 187us) server_negotiation.cc:300] Negotiation successful
0903 08:08:18.857708 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20250903 08:08:18.858536 18548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:18.854415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:43709 (local address 127.12.201.193:47381)
0903 08:08:18.855040 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:18.855074 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:18.855757 (+ 683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:18.856280 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:18.856289 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:18.856370 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:08:18.857043 (+ 673us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0903 08:08:18.857045 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:08:18.857077 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0903 08:08:18.857137 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:08:18.857473 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:08:18.857479 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:08:18.857483 (+ 4us) client_negotiation.cc:769] Sending connection context
0903 08:08:18.857671 (+ 188us) client_negotiation.cc:240] Negotiation successful
0903 08:08:18.857871 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":96,"threads_started":1}
I20250903 08:08:18.858908 13095 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUML90r/build/debug/bin/kudu
/tmp/dist-test-taskUML90r/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756886860253814-13095-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756886860253814-13095-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756886860253814-13095-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756886860253814-13095-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.201.194:0
--local_ip_for_outbound_sockets=127.12.201.194
--webserver_interface=127.12.201.194
--webserver_port=0
--tserver_master_addrs=127.12.201.254:43709
--builtin_ntp_servers=127.12.201.212:42435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20250903 08:08:18.859723 18546 heartbeater.cc:344] Connected to a master server at 127.12.201.254:43709
I20250903 08:08:18.859933 18546 heartbeater.cc:461] Registering TS with master...
I20250903 08:08:18.860316 18546 heartbeater.cc:507] Master 127.12.201.254:43709 requested a full tablet report, sending...
I20250903 08:08:18.861670 18358 ts_manager.cc:194] Registered new tserver with Master: 3e70e903dff24329baceddb7b39a43ba (127.12.201.193:45535)
W20250903 08:08:18.964869 18549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:08:18.965065 18549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:08:18.965119 18549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:08:18.968233 18549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 08:08:18.968281 18549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:08:18.968349 18549 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.201.194
I20250903 08:08:18.972087 18549 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.201.212:42435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756886860253814-13095-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756886860253814-13095-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.201.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756886860253814-13095-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.201.194
--webserver_port=0
--tserver_master_addrs=127.12.201.254:43709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18549
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.201.194
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756886860253814-13095-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 07:50:08 UTC on 5fd53c4cbb9d
build id 7874
I20250903 08:08:18.973047 18549 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:08:18.974112 18549 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:08:18.980491 18554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:18.980509 18555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:08:18.980674 18549 server_base.cc:1047] running on GCE node
W20250903 08:08:18.980561 18557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:08:18.981113 18549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:08:18.981660 18549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:08:18.982820 18549 hybrid_clock.cc:648] HybridClock initialized: now 1756886898982808 us; error 26 us; skew 500 ppm
I20250903 08:08:18.984598 18549 webserver.cc:480] Webserver started at http://127.12.201.194:46113/ using document root <none> and password file <none>
I20250903 08:08:18.985164 18549 fs_manager.cc:362] Metadata directory not provided
I20250903 08:08:18.985222 18549 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:08:18.985409 18549 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:08:18.987138 18549 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756886860253814-13095-0/minicluster-data/ts-1/data/instance:
uuid: "86c393857d824fb48f851f8bc4f730ac"
format_stamp: "Formatted at 2025-09-03 08:08:18 on dist-test-slave-27r9"
server_key: "6de08aed5146f371be8579ded3261fb7"
server_key_iv: "f4100fe0185a311501d6c36be9d33926"
server_key_version: "encryptionkey@0"
I20250903 08:08:18.987610 18549 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756886860253814-13095-0/minicluster-data/ts-1/wal/instance:
uuid: "86c393857d824fb48f851f8bc4f730ac"
format_stamp: "Formatted at 2025-09-03 08:08:18 on dist-test-slave-27r9"
server_key: "6de08aed5146f371be8579ded3261fb7"
server_key_iv: "f4100fe0185a311501d6c36be9d33926"
server_key_version: "encryptionkey@0"
I20250903 08:08:18.990926 18549 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20250903 08:08:18.993096 18563 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:18.994171 18549 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20250903 08:08:18.994273 18549 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756886860253814-13095-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756886860253814-13095-0/minicluster-data/ts-1/wal
uuid: "86c393857d824fb48f851f8bc4f730ac"
format_stamp: "Formatted at 2025-09-03 08:08:18 on dist-test-slave-27r9"
server_key: "6de08aed5146f371be8579ded3261fb7"
server_key_iv: "f4100fe0185a311501d6c36be9d33926"
server_key_version: "encryptionkey@0"
I20250903 08:08:18.994364 18549 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756886860253814-13095-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756886860253814-13095-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756886860253814-13095-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:08:18.998955 18549 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:08:18.999536 18549 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:08:18.999691 18549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:08:19.000185 18549 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 08:08:19.001245 18549 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 08:08:19.001293 18549 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:19.001349 18549 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 08:08:19.001379 18549 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:19.009709 18549 rpc_server.cc:307] RPC server started. Bound to: 127.12.201.194:38109
I20250903 08:08:19.009764 18676 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.201.194:38109 every 8 connection(s)
I20250903 08:08:19.010630 18549 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756886860253814-13095-0/minicluster-data/ts-1/data/info.pb
I20250903 08:08:19.014340 13095 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUML90r/build/debug/bin/kudu as pid 18549
I20250903 08:08:19.014469 13095 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756886860253814-13095-0/minicluster-data/ts-1/wal/instance
I20250903 08:08:19.014757 13095 external_mini_cluster.cc:1468] Setting key 47caa0c77b6cd95b94af53f4f90c359d
I20250903 08:08:19.015797 18398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:19.012329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.194:35851 (local address 127.12.201.254:43709)
0903 08:08:19.012468 (+ 139us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:19.012472 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:19.013324 (+ 852us) server_negotiation.cc:408] Connection header received
0903 08:08:19.014038 (+ 714us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:19.014040 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:19.014079 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:19.014163 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:19.015343 (+ 1180us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:19.015346 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:19.015349 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0903 08:08:19.015361 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:19.015382 (+ 21us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0903 08:08:19.015389 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:19.015484 (+ 95us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:19.015625 (+ 141us) server_negotiation.cc:300] Negotiation successful
0903 08:08:19.015666 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20250903 08:08:19.016470 18679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:19.012584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:43709 (local address 127.12.201.194:35851)
0903 08:08:19.013183 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:19.013216 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:19.013882 (+ 666us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:19.014337 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:19.014345 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:19.014449 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:08:19.015118 (+ 669us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0903 08:08:19.015120 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:08:19.015148 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0903 08:08:19.015199 (+ 51us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:08:19.015480 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:08:19.015488 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:08:19.015494 (+ 6us) client_negotiation.cc:769] Sending connection context
0903 08:08:19.015659 (+ 165us) client_negotiation.cc:240] Negotiation successful
0903 08:08:19.015867 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":95,"threads_started":1}
I20250903 08:08:19.016714 13095 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUML90r/build/debug/bin/kudu
/tmp/dist-test-taskUML90r/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756886860253814-13095-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756886860253814-13095-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756886860253814-13095-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756886860253814-13095-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.201.195:0
--local_ip_for_outbound_sockets=127.12.201.195
--webserver_interface=127.12.201.195
--webserver_port=0
--tserver_master_addrs=127.12.201.254:43709
--builtin_ntp_servers=127.12.201.212:42435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20250903 08:08:19.017570 18677 heartbeater.cc:344] Connected to a master server at 127.12.201.254:43709
I20250903 08:08:19.017813 18677 heartbeater.cc:461] Registering TS with master...
I20250903 08:08:19.018247 18677 heartbeater.cc:507] Master 127.12.201.254:43709 requested a full tablet report, sending...
I20250903 08:08:19.019214 18358 ts_manager.cc:194] Registered new tserver with Master: 86c393857d824fb48f851f8bc4f730ac (127.12.201.194:38109)
W20250903 08:08:19.120791 18680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:08:19.121027 18680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:08:19.121114 18680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:08:19.124408 18680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 08:08:19.124472 18680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:08:19.124594 18680 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.201.195
I20250903 08:08:19.128352 18680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.201.212:42435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756886860253814-13095-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756886860253814-13095-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.201.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756886860253814-13095-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.201.195
--webserver_port=0
--tserver_master_addrs=127.12.201.254:43709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18680
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.201.195
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756886860253814-13095-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 07:50:08 UTC on 5fd53c4cbb9d
build id 7874
I20250903 08:08:19.129462 18680 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:08:19.130626 18680 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:08:19.137188 18688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:19.137228 18686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:19.137218 18685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:08:19.137377 18680 server_base.cc:1047] running on GCE node
I20250903 08:08:19.137811 18680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:08:19.138356 18680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:08:19.139503 18680 hybrid_clock.cc:648] HybridClock initialized: now 1756886899139497 us; error 29 us; skew 500 ppm
I20250903 08:08:19.141233 18680 webserver.cc:480] Webserver started at http://127.12.201.195:41903/ using document root <none> and password file <none>
I20250903 08:08:19.141815 18680 fs_manager.cc:362] Metadata directory not provided
I20250903 08:08:19.141876 18680 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:08:19.142082 18680 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:08:19.143867 18680 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756886860253814-13095-0/minicluster-data/ts-2/data/instance:
uuid: "6fd848fa59ae4a798741e3f68da5b0c6"
format_stamp: "Formatted at 2025-09-03 08:08:19 on dist-test-slave-27r9"
server_key: "e6e04d8d1c44d867350b59337b22c518"
server_key_iv: "cc3c28ff0b6577b63a7e45ca8d8e3c81"
server_key_version: "encryptionkey@0"
I20250903 08:08:19.144356 18680 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756886860253814-13095-0/minicluster-data/ts-2/wal/instance:
uuid: "6fd848fa59ae4a798741e3f68da5b0c6"
format_stamp: "Formatted at 2025-09-03 08:08:19 on dist-test-slave-27r9"
server_key: "e6e04d8d1c44d867350b59337b22c518"
server_key_iv: "cc3c28ff0b6577b63a7e45ca8d8e3c81"
server_key_version: "encryptionkey@0"
I20250903 08:08:19.147789 18680 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20250903 08:08:19.149979 18694 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:19.151086 18680 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250903 08:08:19.151190 18680 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756886860253814-13095-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756886860253814-13095-0/minicluster-data/ts-2/wal
uuid: "6fd848fa59ae4a798741e3f68da5b0c6"
format_stamp: "Formatted at 2025-09-03 08:08:19 on dist-test-slave-27r9"
server_key: "e6e04d8d1c44d867350b59337b22c518"
server_key_iv: "cc3c28ff0b6577b63a7e45ca8d8e3c81"
server_key_version: "encryptionkey@0"
I20250903 08:08:19.151314 18680 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756886860253814-13095-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756886860253814-13095-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756886860253814-13095-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:08:19.156109 18680 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:08:19.156641 18680 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:08:19.156775 18680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:08:19.157227 18680 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 08:08:19.158087 18680 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 08:08:19.158131 18680 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:19.158169 18680 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 08:08:19.158190 18680 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:19.167028 18680 rpc_server.cc:307] RPC server started. Bound to: 127.12.201.195:42727
I20250903 08:08:19.167084 18807 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.201.195:42727 every 8 connection(s)
I20250903 08:08:19.168072 18680 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756886860253814-13095-0/minicluster-data/ts-2/data/info.pb
I20250903 08:08:19.172364 13095 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUML90r/build/debug/bin/kudu as pid 18680
I20250903 08:08:19.172492 13095 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756886860253814-13095-0/minicluster-data/ts-2/wal/instance
I20250903 08:08:19.172752 13095 external_mini_cluster.cc:1468] Setting key ccca67a7366ef24d1f2173195108ef32
I20250903 08:08:19.173687 18398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:19.169920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.195:38057 (local address 127.12.201.254:43709)
0903 08:08:19.170090 (+ 170us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:19.170094 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:19.171014 (+ 920us) server_negotiation.cc:408] Connection header received
0903 08:08:19.171801 (+ 787us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:19.171804 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:19.171843 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:19.171917 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:19.173123 (+ 1206us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:19.173127 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:19.173130 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0903 08:08:19.173143 (+ 13us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:19.173169 (+ 26us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0903 08:08:19.173181 (+ 12us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:19.173291 (+ 110us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:19.173514 (+ 223us) server_negotiation.cc:300] Negotiation successful
0903 08:08:19.173556 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20250903 08:08:19.174479 18810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:19.170211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:43709 (local address 127.12.201.195:38057)
0903 08:08:19.170854 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:19.170895 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:19.171625 (+ 730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:19.172068 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:19.172078 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:19.172185 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:08:19.172846 (+ 661us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0903 08:08:19.172848 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:08:19.172890 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0903 08:08:19.172957 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:08:19.173318 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:08:19.173328 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:08:19.173335 (+ 7us) client_negotiation.cc:769] Sending connection context
0903 08:08:19.173542 (+ 207us) client_negotiation.cc:240] Negotiation successful
0903 08:08:19.173767 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":102,"threads_started":1}
I20250903 08:08:19.175560 18808 heartbeater.cc:344] Connected to a master server at 127.12.201.254:43709
I20250903 08:08:19.175804 18808 heartbeater.cc:461] Registering TS with master...
I20250903 08:08:19.176218 18808 heartbeater.cc:507] Master 127.12.201.254:43709 requested a full tablet report, sending...
I20250903 08:08:19.177175 18358 ts_manager.cc:194] Registered new tserver with Master: 6fd848fa59ae4a798741e3f68da5b0c6 (127.12.201.195:42727)
I20250903 08:08:19.186632 13095 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250903 08:08:19.190094 18398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:19.188491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51068 (local address 127.12.201.254:43709)
0903 08:08:19.188666 (+ 175us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:19.188671 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:19.188772 (+ 101us) server_negotiation.cc:408] Connection header received
0903 08:08:19.188896 (+ 124us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:19.188899 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:19.188938 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:19.189007 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:19.189719 (+ 712us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:19.189722 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:19.189724 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0903 08:08:19.189733 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:19.189750 (+ 17us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0903 08:08:19.189757 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:19.189832 (+ 75us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:19.189923 (+ 91us) server_negotiation.cc:300] Negotiation successful
0903 08:08:19.189960 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20250903 08:08:19.192667 18358 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51068:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 08:08:19.194952 18358 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 08:08:19.205616 18819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:19.202538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.194:38109 (local address 127.0.0.1:46918)
0903 08:08:19.203229 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:19.203272 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:19.203392 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:19.204073 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:19.204079 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:19.204144 (+ 65us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:08:19.204768 (+ 624us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0903 08:08:19.204772 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:08:19.204805 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0903 08:08:19.204856 (+ 51us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:08:19.205240 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:08:19.205251 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:08:19.205261 (+ 10us) client_negotiation.cc:769] Sending connection context
0903 08:08:19.205357 (+ 96us) client_negotiation.cc:240] Negotiation successful
0903 08:08:19.205425 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"spinlock_wait_cycles":22784,"thread_start_us":61,"threads_started":1}
I20250903 08:08:19.206336 18823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:19.203177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.193:45535 (local address 127.0.0.1:38164)
0903 08:08:19.204643 (+ 1466us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:19.204668 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:19.204744 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:19.205206 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:19.205210 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:19.205224 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:08:19.205804 (+ 580us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0903 08:08:19.205806 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:08:19.205820 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0903 08:08:19.205842 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:08:19.206151 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:08:19.206154 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:08:19.206156 (+ 2us) client_negotiation.cc:769] Sending connection context
0903 08:08:19.206213 (+ 57us) client_negotiation.cc:240] Negotiation successful
0903 08:08:19.206240 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1400,"thread_start_us":76,"threads_started":1}
I20250903 08:08:19.206425 18820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:19.202708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46918 (local address 127.12.201.194:38109)
0903 08:08:19.203626 (+ 918us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:19.203631 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:19.203651 (+ 20us) server_negotiation.cc:408] Connection header received
0903 08:08:19.203696 (+ 45us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:19.203702 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:19.203874 (+ 172us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:19.204027 (+ 153us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:19.204999 (+ 972us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:19.205005 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:19.205034 (+ 29us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0903 08:08:19.205047 (+ 13us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:19.205109 (+ 62us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0903 08:08:19.205124 (+ 15us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:19.205831 (+ 707us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:19.205878 (+ 47us) server_negotiation.cc:300] Negotiation successful
0903 08:08:19.205951 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":830,"thread_start_us":86,"threads_started":1}
I20250903 08:08:19.207067 18821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:19.202872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.195:42727 (local address 127.0.0.1:60686)
0903 08:08:19.203228 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:19.203272 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:19.203392 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:19.204807 (+ 1415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:19.204812 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:19.204824 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:08:19.205402 (+ 578us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0903 08:08:19.205404 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:08:19.205417 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0903 08:08:19.205448 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:08:19.206808 (+ 1360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:08:19.206812 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:08:19.206815 (+ 3us) client_negotiation.cc:769] Sending connection context
0903 08:08:19.206898 (+ 83us) client_negotiation.cc:240] Negotiation successful
0903 08:08:19.206933 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"spinlock_wait_cycles":7424,"thread_start_us":67,"threads_started":1}
I20250903 08:08:19.207491 18822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:19.203078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60686 (local address 127.12.201.195:42727)
0903 08:08:19.203733 (+ 655us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:19.203740 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:19.203763 (+ 23us) server_negotiation.cc:408] Connection header received
0903 08:08:19.203819 (+ 56us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:19.203824 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:19.203990 (+ 166us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:19.204165 (+ 175us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:19.205597 (+ 1432us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:19.205603 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:19.205612 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0903 08:08:19.205623 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:19.205675 (+ 52us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0903 08:08:19.205687 (+ 12us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:19.206828 (+ 1141us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:19.206942 (+ 114us) server_negotiation.cc:300] Negotiation successful
0903 08:08:19.207028 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":140,"threads_started":1}
I20250903 08:08:19.207926 18824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:19.203573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38164 (local address 127.12.201.193:45535)
0903 08:08:19.203889 (+ 316us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:19.203896 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:19.204830 (+ 934us) server_negotiation.cc:408] Connection header received
0903 08:08:19.204891 (+ 61us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:19.204897 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:19.205052 (+ 155us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:19.205172 (+ 120us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:19.205963 (+ 791us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:19.205968 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:19.205975 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0903 08:08:19.205987 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:19.206049 (+ 62us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0903 08:08:19.206060 (+ 11us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:19.206182 (+ 122us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:19.206229 (+ 47us) server_negotiation.cc:300] Negotiation successful
0903 08:08:19.206304 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":108,"threads_started":1}
I20250903 08:08:19.209496 18611 tablet_service.cc:1468] Processing CreateTablet for tablet eef237dabb34411bae7d530f0fb6bc8d (DEFAULT_TABLE table=test-table [id=d158ee666ddc4bd7a7237c598560e568]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:08:19.209853 18742 tablet_service.cc:1468] Processing CreateTablet for tablet eef237dabb34411bae7d530f0fb6bc8d (DEFAULT_TABLE table=test-table [id=d158ee666ddc4bd7a7237c598560e568]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:08:19.210573 18611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eef237dabb34411bae7d530f0fb6bc8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:08:19.210356 18480 tablet_service.cc:1468] Processing CreateTablet for tablet eef237dabb34411bae7d530f0fb6bc8d (DEFAULT_TABLE table=test-table [id=d158ee666ddc4bd7a7237c598560e568]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:08:19.210691 18742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eef237dabb34411bae7d530f0fb6bc8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:08:19.211175 18480 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eef237dabb34411bae7d530f0fb6bc8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:08:19.215834 18825 tablet_bootstrap.cc:492] T eef237dabb34411bae7d530f0fb6bc8d P 6fd848fa59ae4a798741e3f68da5b0c6: Bootstrap starting.
I20250903 08:08:19.217057 18826 tablet_bootstrap.cc:492] T eef237dabb34411bae7d530f0fb6bc8d P 3e70e903dff24329baceddb7b39a43ba: Bootstrap starting.
I20250903 08:08:19.217806 18825 tablet_bootstrap.cc:654] T eef237dabb34411bae7d530f0fb6bc8d P 6fd848fa59ae4a798741e3f68da5b0c6: Neither blocks nor log segments found. Creating new log.
I20250903 08:08:19.218502 18825 log.cc:826] T eef237dabb34411bae7d530f0fb6bc8d P 6fd848fa59ae4a798741e3f68da5b0c6: Log is configured to *not* fsync() on all Append() calls
I20250903 08:08:19.218607 18826 tablet_bootstrap.cc:654] T eef237dabb34411bae7d530f0fb6bc8d P 3e70e903dff24329baceddb7b39a43ba: Neither blocks nor log segments found. Creating new log.
I20250903 08:08:19.219349 18827 tablet_bootstrap.cc:492] T eef237dabb34411bae7d530f0fb6bc8d P 86c393857d824fb48f851f8bc4f730ac: Bootstrap starting.
I20250903 08:08:19.219388 18826 log.cc:826] T eef237dabb34411bae7d530f0fb6bc8d P 3e70e903dff24329baceddb7b39a43ba: Log is configured to *not* fsync() on all Append() calls
I20250903 08:08:19.220820 18826 tablet_bootstrap.cc:492] T eef237dabb34411bae7d530f0fb6bc8d P 3e70e903dff24329baceddb7b39a43ba: No bootstrap required, opened a new log
I20250903 08:08:19.220839 18827 tablet_bootstrap.cc:654] T eef237dabb34411bae7d530f0fb6bc8d P 86c393857d824fb48f851f8bc4f730ac: Neither blocks nor log segments found. Creating new log.
I20250903 08:08:19.220968 18826 ts_tablet_manager.cc:1397] T eef237dabb34411bae7d530f0fb6bc8d P 3e70e903dff24329baceddb7b39a43ba: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250903 08:08:19.221446 18827 log.cc:826] T eef237dabb34411bae7d530f0fb6bc8d P 86c393857d824fb48f851f8bc4f730ac: Log is configured to *not* fsync() on all Append() calls
I20250903 08:08:19.222616 18825 tablet_bootstrap.cc:492] T eef237dabb34411bae7d530f0fb6bc8d P 6fd848fa59ae4a798741e3f68da5b0c6: No bootstrap required, opened a new log
I20250903 08:08:19.222780 18825 ts_tablet_manager.cc:1397] T eef237dabb34411bae7d530f0fb6bc8d P 6fd848fa59ae4a798741e3f68da5b0c6: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.001s
I20250903 08:08:19.223380 18826 raft_consensus.cc:357] T eef237dabb34411bae7d530f0fb6bc8d P 3e70e903dff24329baceddb7b39a43ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86c393857d824fb48f851f8bc4f730ac" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 38109 } } peers { permanent_uuid: "3e70e903dff24329baceddb7b39a43ba" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 45535 } } peers { permanent_uuid: "6fd848fa59ae4a798741e3f68da5b0c6" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 42727 } }
I20250903 08:08:19.223552 18826 raft_consensus.cc:383] T eef237dabb34411bae7d530f0fb6bc8d P 3e70e903dff24329baceddb7b39a43ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:08:19.223611 18826 raft_consensus.cc:738] T eef237dabb34411bae7d530f0fb6bc8d P 3e70e903dff24329baceddb7b39a43ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e70e903dff24329baceddb7b39a43ba, State: Initialized, Role: FOLLOWER
I20250903 08:08:19.224045 18826 consensus_queue.cc:260] T eef237dabb34411bae7d530f0fb6bc8d P 3e70e903dff24329baceddb7b39a43ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86c393857d824fb48f851f8bc4f730ac" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 38109 } } peers { permanent_uuid: "3e70e903dff24329baceddb7b39a43ba" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 45535 } } peers { permanent_uuid: "6fd848fa59ae4a798741e3f68da5b0c6" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 42727 } }
I20250903 08:08:19.224756 18826 ts_tablet_manager.cc:1428] T eef237dabb34411bae7d530f0fb6bc8d P 3e70e903dff24329baceddb7b39a43ba: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250903 08:08:19.225099 18546 heartbeater.cc:499] Master 127.12.201.254:43709 was elected leader, sending a full tablet report...
I20250903 08:08:19.225247 18825 raft_consensus.cc:357] T eef237dabb34411bae7d530f0fb6bc8d P 6fd848fa59ae4a798741e3f68da5b0c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86c393857d824fb48f851f8bc4f730ac" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 38109 } } peers { permanent_uuid: "3e70e903dff24329baceddb7b39a43ba" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 45535 } } peers { permanent_uuid: "6fd848fa59ae4a798741e3f68da5b0c6" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 42727 } }
I20250903 08:08:19.225442 18825 raft_consensus.cc:383] T eef237dabb34411bae7d530f0fb6bc8d P 6fd848fa59ae4a798741e3f68da5b0c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:08:19.225495 18825 raft_consensus.cc:738] T eef237dabb34411bae7d530f0fb6bc8d P 6fd848fa59ae4a798741e3f68da5b0c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fd848fa59ae4a798741e3f68da5b0c6, State: Initialized, Role: FOLLOWER
I20250903 08:08:19.225880 18825 consensus_queue.cc:260] T eef237dabb34411bae7d530f0fb6bc8d P 6fd848fa59ae4a798741e3f68da5b0c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86c393857d824fb48f851f8bc4f730ac" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 38109 } } peers { permanent_uuid: "3e70e903dff24329baceddb7b39a43ba" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 45535 } } peers { permanent_uuid: "6fd848fa59ae4a798741e3f68da5b0c6" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 42727 } }
I20250903 08:08:19.226009 18827 tablet_bootstrap.cc:492] T eef237dabb34411bae7d530f0fb6bc8d P 86c393857d824fb48f851f8bc4f730ac: No bootstrap required, opened a new log
I20250903 08:08:19.226167 18827 ts_tablet_manager.cc:1397] T eef237dabb34411bae7d530f0fb6bc8d P 86c393857d824fb48f851f8bc4f730ac: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20250903 08:08:19.226577 18808 heartbeater.cc:499] Master 127.12.201.254:43709 was elected leader, sending a full tablet report...
I20250903 08:08:19.226727 18825 ts_tablet_manager.cc:1428] T eef237dabb34411bae7d530f0fb6bc8d P 6fd848fa59ae4a798741e3f68da5b0c6: Time spent starting tablet: real 0.004s user 0.001s sys 0.003s
I20250903 08:08:19.229094 18827 raft_consensus.cc:357] T eef237dabb34411bae7d530f0fb6bc8d P 86c393857d824fb48f851f8bc4f730ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86c393857d824fb48f851f8bc4f730ac" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 38109 } } peers { permanent_uuid: "3e70e903dff24329baceddb7b39a43ba" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 45535 } } peers { permanent_uuid: "6fd848fa59ae4a798741e3f68da5b0c6" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 42727 } }
I20250903 08:08:19.229331 18827 raft_consensus.cc:383] T eef237dabb34411bae7d530f0fb6bc8d P 86c393857d824fb48f851f8bc4f730ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:08:19.229394 18827 raft_consensus.cc:738] T eef237dabb34411bae7d530f0fb6bc8d P 86c393857d824fb48f851f8bc4f730ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86c393857d824fb48f851f8bc4f730ac, State: Initialized, Role: FOLLOWER
I20250903 08:08:19.229815 18827 consensus_queue.cc:260] T eef237dabb34411bae7d530f0fb6bc8d P 86c393857d824fb48f851f8bc4f730ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86c393857d824fb48f851f8bc4f730ac" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 38109 } } peers { permanent_uuid: "3e70e903dff24329baceddb7b39a43ba" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 45535 } } peers { permanent_uuid: "6fd848fa59ae4a798741e3f68da5b0c6" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 42727 } }
I20250903 08:08:19.230466 18677 heartbeater.cc:499] Master 127.12.201.254:43709 was elected leader, sending a full tablet report...
I20250903 08:08:19.230674 18827 ts_tablet_manager.cc:1428] T eef237dabb34411bae7d530f0fb6bc8d P 86c393857d824fb48f851f8bc4f730ac: Time spent starting tablet: real 0.004s user 0.000s sys 0.005s
W20250903 08:08:19.261509 18678 tablet.cc:2378] T eef237dabb34411bae7d530f0fb6bc8d P 86c393857d824fb48f851f8bc4f730ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250903 08:08:19.353407 18547 tablet.cc:2378] T eef237dabb34411bae7d530f0fb6bc8d P 3e70e903dff24329baceddb7b39a43ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250903 08:08:19.419095 18809 tablet.cc:2378] T eef237dabb34411bae7d530f0fb6bc8d P 6fd848fa59ae4a798741e3f68da5b0c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250903 08:08:19.510790 18833 raft_consensus.cc:491] T eef237dabb34411bae7d530f0fb6bc8d P 86c393857d824fb48f851f8bc4f730ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 08:08:19.511045 18833 raft_consensus.cc:513] T eef237dabb34411bae7d530f0fb6bc8d P 86c393857d824fb48f851f8bc4f730ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86c393857d824fb48f851f8bc4f730ac" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 38109 } } peers { permanent_uuid: "3e70e903dff24329baceddb7b39a43ba" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 45535 } } peers { permanent_uuid: "6fd848fa59ae4a798741e3f68da5b0c6" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 42727 } }
I20250903 08:08:19.512199 18833 leader_election.cc:290] T eef237dabb34411bae7d530f0fb6bc8d P 86c393857d824fb48f851f8bc4f730ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e70e903dff24329baceddb7b39a43ba (127.12.201.193:45535), 6fd848fa59ae4a798741e3f68da5b0c6 (127.12.201.195:42727)
I20250903 08:08:19.514118 18824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:19.512633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.194:56715 (local address 127.12.201.193:45535)
0903 08:08:19.512751 (+ 118us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:19.512756 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:19.512773 (+ 17us) server_negotiation.cc:408] Connection header received
0903 08:08:19.512824 (+ 51us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:19.512826 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:19.512867 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:19.512958 (+ 91us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:19.513728 (+ 770us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:19.513731 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:19.513734 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0903 08:08:19.513746 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:19.513765 (+ 19us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0903 08:08:19.513773 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:19.513846 (+ 73us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:19.513965 (+ 119us) server_negotiation.cc:300] Negotiation successful
0903 08:08:19.514002 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20250903 08:08:19.514199 18679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:19.512434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.193:45535 (local address 127.12.201.194:56715)
0903 08:08:19.512597 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:19.512614 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:19.512696 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:19.512989 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:19.512991 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:19.513003 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:08:19.513546 (+ 543us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0903 08:08:19.513547 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:08:19.513557 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0903 08:08:19.513578 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:08:19.513860 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:08:19.513865 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:08:19.513868 (+ 3us) client_negotiation.cc:769] Sending connection context
0903 08:08:19.513966 (+ 98us) client_negotiation.cc:240] Negotiation successful
0903 08:08:19.514009 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20250903 08:08:19.514839 18834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:19.512478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.195:42727 (local address 127.12.201.194:44649)
0903 08:08:19.512803 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:19.512818 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:19.512878 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:19.513157 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:19.513162 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:19.513181 (+ 19us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:08:19.514210 (+ 1029us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0903 08:08:19.514214 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:08:19.514227 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0903 08:08:19.514250 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:08:19.514598 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:08:19.514604 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:08:19.514611 (+ 7us) client_negotiation.cc:769] Sending connection context
0903 08:08:19.514687 (+ 76us) client_negotiation.cc:240] Negotiation successful
0903 08:08:19.514713 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":83,"threads_started":1}
I20250903 08:08:19.514860 18822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:19.512617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.194:44649 (local address 127.12.201.195:42727)
0903 08:08:19.512758 (+ 141us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:19.512763 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:19.512929 (+ 166us) server_negotiation.cc:408] Connection header received
0903 08:08:19.512974 (+ 45us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:19.512977 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:19.513018 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:19.513086 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:19.514432 (+ 1346us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:19.514436 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:19.514439 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0903 08:08:19.514470 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:19.514489 (+ 19us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0903 08:08:19.514498 (+ 9us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:19.514585 (+ 87us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:19.514709 (+ 124us) server_negotiation.cc:300] Negotiation successful
0903 08:08:19.514748 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20250903 08:08:19.514905 18500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eef237dabb34411bae7d530f0fb6bc8d" candidate_uuid: "86c393857d824fb48f851f8bc4f730ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e70e903dff24329baceddb7b39a43ba" is_pre_election: true
I20250903 08:08:19.515208 18500 raft_consensus.cc:2466] T eef237dabb34411bae7d530f0fb6bc8d P 3e70e903dff24329baceddb7b39a43ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86c393857d824fb48f851f8bc4f730ac in term 0.
I20250903 08:08:19.515470 18762 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eef237dabb34411bae7d530f0fb6bc8d" candidate_uuid: "86c393857d824fb48f851f8bc4f730ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fd848fa59ae4a798741e3f68da5b0c6" is_pre_election: true
I20250903 08:08:19.515695 18567 leader_election.cc:304] T eef237dabb34411bae7d530f0fb6bc8d P 86c393857d824fb48f851f8bc4f730ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e70e903dff24329baceddb7b39a43ba, 86c393857d824fb48f851f8bc4f730ac; no voters:
I20250903 08:08:19.515728 18762 raft_consensus.cc:2466] T eef237dabb34411bae7d530f0fb6bc8d P 6fd848fa59ae4a798741e3f68da5b0c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86c393857d824fb48f851f8bc4f730ac in term 0.
I20250903 08:08:19.515910 18833 raft_consensus.cc:2802] T eef237dabb34411bae7d530f0fb6bc8d P 86c393857d824fb48f851f8bc4f730ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 08:08:19.515983 18833 raft_consensus.cc:491] T eef237dabb34411bae7d530f0fb6bc8d P 86c393857d824fb48f851f8bc4f730ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 08:08:19.516038 18833 raft_consensus.cc:3058] T eef237dabb34411bae7d530f0fb6bc8d P 86c393857d824fb48f851f8bc4f730ac [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:08:19.516973 18833 raft_consensus.cc:513] T eef237dabb34411bae7d530f0fb6bc8d P 86c393857d824fb48f851f8bc4f730ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86c393857d824fb48f851f8bc4f730ac" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 38109 } } peers { permanent_uuid: "3e70e903dff24329baceddb7b39a43ba" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 45535 } } peers { permanent_uuid: "6fd848fa59ae4a798741e3f68da5b0c6" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 42727 } }
I20250903 08:08:19.517304 18833 leader_election.cc:290] T eef237dabb34411bae7d530f0fb6bc8d P 86c393857d824fb48f851f8bc4f730ac [CANDIDATE]: Term 1 election: Requested vote from peers 3e70e903dff24329baceddb7b39a43ba (127.12.201.193:45535), 6fd848fa59ae4a798741e3f68da5b0c6 (127.12.201.195:42727)
I20250903 08:08:19.517607 18500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eef237dabb34411bae7d530f0fb6bc8d" candidate_uuid: "86c393857d824fb48f851f8bc4f730ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e70e903dff24329baceddb7b39a43ba"
I20250903 08:08:19.517638 18762 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eef237dabb34411bae7d530f0fb6bc8d" candidate_uuid: "86c393857d824fb48f851f8bc4f730ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fd848fa59ae4a798741e3f68da5b0c6"
I20250903 08:08:19.517719 18500 raft_consensus.cc:3058] T eef237dabb34411bae7d530f0fb6bc8d P 3e70e903dff24329baceddb7b39a43ba [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:08:19.517737 18762 raft_consensus.cc:3058] T eef237dabb34411bae7d530f0fb6bc8d P 6fd848fa59ae4a798741e3f68da5b0c6 [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:08:19.518584 18500 raft_consensus.cc:2466] T eef237dabb34411bae7d530f0fb6bc8d P 3e70e903dff24329baceddb7b39a43ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86c393857d824fb48f851f8bc4f730ac in term 1.
I20250903 08:08:19.518852 18567 leader_election.cc:304] T eef237dabb34411bae7d530f0fb6bc8d P 86c393857d824fb48f851f8bc4f730ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e70e903dff24329baceddb7b39a43ba, 86c393857d824fb48f851f8bc4f730ac; no voters:
I20250903 08:08:19.519032 18833 raft_consensus.cc:2802] T eef237dabb34411bae7d530f0fb6bc8d P 86c393857d824fb48f851f8bc4f730ac [term 1 FOLLOWER]: Leader election won for term 1
I20250903 08:08:19.519097 18762 raft_consensus.cc:2466] T eef237dabb34411bae7d530f0fb6bc8d P 6fd848fa59ae4a798741e3f68da5b0c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86c393857d824fb48f851f8bc4f730ac in term 1.
I20250903 08:08:19.519233 18833 raft_consensus.cc:695] T eef237dabb34411bae7d530f0fb6bc8d P 86c393857d824fb48f851f8bc4f730ac [term 1 LEADER]: Becoming Leader. State: Replica: 86c393857d824fb48f851f8bc4f730ac, State: Running, Role: LEADER
I20250903 08:08:19.519529 18833 consensus_queue.cc:237] T eef237dabb34411bae7d530f0fb6bc8d P 86c393857d824fb48f851f8bc4f730ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86c393857d824fb48f851f8bc4f730ac" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 38109 } } peers { permanent_uuid: "3e70e903dff24329baceddb7b39a43ba" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 45535 } } peers { permanent_uuid: "6fd848fa59ae4a798741e3f68da5b0c6" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 42727 } }
I20250903 08:08:19.522369 18357 catalog_manager.cc:5582] T eef237dabb34411bae7d530f0fb6bc8d P 86c393857d824fb48f851f8bc4f730ac reported cstate change: term changed from 0 to 1, leader changed from <none> to 86c393857d824fb48f851f8bc4f730ac (127.12.201.194). New cstate: current_term: 1 leader_uuid: "86c393857d824fb48f851f8bc4f730ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86c393857d824fb48f851f8bc4f730ac" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 38109 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e70e903dff24329baceddb7b39a43ba" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 45535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6fd848fa59ae4a798741e3f68da5b0c6" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 42727 } health_report { overall_health: UNKNOWN } } }
I20250903 08:08:19.574528 18820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:19.572935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46934 (local address 127.12.201.194:38109)
0903 08:08:19.573097 (+ 162us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:19.573103 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:19.573118 (+ 15us) server_negotiation.cc:408] Connection header received
0903 08:08:19.573210 (+ 92us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:19.573213 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:19.573268 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:19.573341 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:19.574108 (+ 767us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:19.574111 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:19.574114 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0903 08:08:19.574126 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:19.574145 (+ 19us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0903 08:08:19.574153 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:19.574234 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:19.574349 (+ 115us) server_negotiation.cc:300] Negotiation successful
0903 08:08:19.574388 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20250903 08:08:19.579854 18762 raft_consensus.cc:1273] T eef237dabb34411bae7d530f0fb6bc8d P 6fd848fa59ae4a798741e3f68da5b0c6 [term 1 FOLLOWER]: Refusing update from remote peer 86c393857d824fb48f851f8bc4f730ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 08:08:19.579854 18500 raft_consensus.cc:1273] T eef237dabb34411bae7d530f0fb6bc8d P 3e70e903dff24329baceddb7b39a43ba [term 1 FOLLOWER]: Refusing update from remote peer 86c393857d824fb48f851f8bc4f730ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 08:08:19.580502 18835 consensus_queue.cc:1035] T eef237dabb34411bae7d530f0fb6bc8d P 86c393857d824fb48f851f8bc4f730ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fd848fa59ae4a798741e3f68da5b0c6" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 42727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 08:08:19.580657 18833 consensus_queue.cc:1035] T eef237dabb34411bae7d530f0fb6bc8d P 86c393857d824fb48f851f8bc4f730ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e70e903dff24329baceddb7b39a43ba" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 45535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 08:08:19.587067 18838 mvcc.cc:204] Tried to move back new op lower bound from 7196208740672552960 to 7196208740434206720. Current Snapshot: MvccSnapshot[applied={T|T < 7196208740672552960}]
I20250903 08:08:19.590620 18840 mvcc.cc:204] Tried to move back new op lower bound from 7196208740672552960 to 7196208740434206720. Current Snapshot: MvccSnapshot[applied={T|T < 7196208740672552960 or (T in {7196208740672552960})}]
I20250903 08:08:19.595669 18357 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51068:
table { table_name: "test-table" } modify_external_catalogs: true
I20250903 08:08:19.595829 18357 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51068:
table { table_name: "test-table" } modify_external_catalogs: true
I20250903 08:08:19.598186 18357 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c1056cbdf1a84732889259c48109686d: Sending DeleteTablet for 3 replicas of tablet eef237dabb34411bae7d530f0fb6bc8d
I20250903 08:08:19.598827 18611 tablet_service.cc:1515] Processing DeleteTablet for tablet eef237dabb34411bae7d530f0fb6bc8d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 08:08:19 UTC) from {username='slave'} at 127.0.0.1:46918
I20250903 08:08:19.598827 18742 tablet_service.cc:1515] Processing DeleteTablet for tablet eef237dabb34411bae7d530f0fb6bc8d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 08:08:19 UTC) from {username='slave'} at 127.0.0.1:60686
I20250903 08:08:19.599344 18848 tablet_replica.cc:331] T eef237dabb34411bae7d530f0fb6bc8d P 86c393857d824fb48f851f8bc4f730ac: stopping tablet replica
I20250903 08:08:19.599340 18480 tablet_service.cc:1515] Processing DeleteTablet for tablet eef237dabb34411bae7d530f0fb6bc8d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 08:08:19 UTC) from {username='slave'} at 127.0.0.1:38164
I20250903 08:08:19.599691 18848 raft_consensus.cc:2241] T eef237dabb34411bae7d530f0fb6bc8d P 86c393857d824fb48f851f8bc4f730ac [term 1 LEADER]: Raft consensus shutting down.
I20250903 08:08:19.599807 13095 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUML90r/build/debug/bin/kudu with pid 18418
I20250903 08:08:19.601382 18848 raft_consensus.cc:2270] T eef237dabb34411bae7d530f0fb6bc8d P 86c393857d824fb48f851f8bc4f730ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 08:08:19.602591 18848 ts_tablet_manager.cc:1905] T eef237dabb34411bae7d530f0fb6bc8d P 86c393857d824fb48f851f8bc4f730ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 08:08:19.604362 18850 tablet_replica.cc:331] T eef237dabb34411bae7d530f0fb6bc8d P 6fd848fa59ae4a798741e3f68da5b0c6: stopping tablet replica
I20250903 08:08:19.604697 18850 raft_consensus.cc:2241] T eef237dabb34411bae7d530f0fb6bc8d P 6fd848fa59ae4a798741e3f68da5b0c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 08:08:19.605067 18850 raft_consensus.cc:2270] T eef237dabb34411bae7d530f0fb6bc8d P 6fd848fa59ae4a798741e3f68da5b0c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 08:08:19.605192 18848 ts_tablet_manager.cc:1918] T eef237dabb34411bae7d530f0fb6bc8d P 86c393857d824fb48f851f8bc4f730ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250903 08:08:19.605285 18848 log.cc:1199] T eef237dabb34411bae7d530f0fb6bc8d P 86c393857d824fb48f851f8bc4f730ac: Deleting WAL directory at /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756886860253814-13095-0/minicluster-data/ts-1/wal/wals/eef237dabb34411bae7d530f0fb6bc8d
I20250903 08:08:19.605633 18848 ts_tablet_manager.cc:1939] T eef237dabb34411bae7d530f0fb6bc8d P 86c393857d824fb48f851f8bc4f730ac: Deleting consensus metadata
I20250903 08:08:19.606659 18850 ts_tablet_manager.cc:1905] T eef237dabb34411bae7d530f0fb6bc8d P 6fd848fa59ae4a798741e3f68da5b0c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 08:08:19.606776 18345 catalog_manager.cc:4928] TS 86c393857d824fb48f851f8bc4f730ac (127.12.201.194:38109): tablet eef237dabb34411bae7d530f0fb6bc8d (table test-table [id=d158ee666ddc4bd7a7237c598560e568]) successfully deleted
W20250903 08:08:19.607043 18345 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.201.193:45535 (error 108)
I20250903 08:08:19.607102 13095 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUML90r/build/debug/bin/kudu with pid 18549
I20250903 08:08:19.608495 18821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:19.608118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.193:45535 (local address 127.0.0.1:38178)
0903 08:08:19.608325 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:19.608390 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.201.193:45535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
W20250903 08:08:19.608640 18345 catalog_manager.cc:4671] TS 3e70e903dff24329baceddb7b39a43ba (127.12.201.193:45535): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet eef237dabb34411bae7d530f0fb6bc8d: Network error: Client connection negotiation failed: client connection to 127.12.201.193:45535: connect: Connection refused (error 111)
I20250903 08:08:19.611419 18850 ts_tablet_manager.cc:1918] T eef237dabb34411bae7d530f0fb6bc8d P 6fd848fa59ae4a798741e3f68da5b0c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250903 08:08:19.611513 18850 log.cc:1199] T eef237dabb34411bae7d530f0fb6bc8d P 6fd848fa59ae4a798741e3f68da5b0c6: Deleting WAL directory at /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756886860253814-13095-0/minicluster-data/ts-2/wal/wals/eef237dabb34411bae7d530f0fb6bc8d
I20250903 08:08:19.611833 18850 ts_tablet_manager.cc:1939] T eef237dabb34411bae7d530f0fb6bc8d P 6fd848fa59ae4a798741e3f68da5b0c6: Deleting consensus metadata
I20250903 08:08:19.612942 18344 catalog_manager.cc:4928] TS 6fd848fa59ae4a798741e3f68da5b0c6 (127.12.201.195:42727): tablet eef237dabb34411bae7d530f0fb6bc8d (table test-table [id=d158ee666ddc4bd7a7237c598560e568]) successfully deleted
I20250903 08:08:19.614018 13095 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUML90r/build/debug/bin/kudu with pid 18680
I20250903 08:08:19.619474 13095 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUML90r/build/debug/bin/kudu with pid 18327
2025-09-03T08:08:19Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1160 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20250903 08:08:19.640893 13095 mini_oidc.cc:150] Starting JWKS server
I20250903 08:08:19.642282 13095 webserver.cc:480] Webserver started at https://127.0.0.1:32771/ using document root <none> and password file <none>
I20250903 08:08:19.642436 13095 mini_oidc.cc:186] Starting OIDC Discovery server
I20250903 08:08:19.642660 13095 webserver.cc:480] Webserver started at http://127.0.0.1:44087/ using document root <none> and password file <none>
2025-09-03T08:08:19Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-03T08:08:19Z Disabled control of system clock
I20250903 08:08:19.650276 13095 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUML90r/build/debug/bin/kudu
/tmp/dist-test-taskUML90r/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1756886860253814-13095-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1756886860253814-13095-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1756886860253814-13095-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1756886860253814-13095-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.201.254:35831
--webserver_interface=127.12.201.254
--webserver_port=0
--builtin_ntp_servers=127.12.201.212:35951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.201.254:35831
--enable_jwt_token_auth=true
--jwks_url=https://localhost:32771/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1756886860253814-13095-0/testchainca.cert with env {}
W20250903 08:08:19.756588 18857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:08:19.756817 18857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:08:19.756868 18857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:08:19.760416 18857 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250903 08:08:19.760479 18857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 08:08:19.760496 18857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:08:19.760516 18857 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250903 08:08:19.760535 18857 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20250903 08:08:19.760568 18857 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20250903 08:08:19.760596 18857 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:32771/jwks.json
I20250903 08:08:19.765242 18857 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.201.212:35951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1756886860253814-13095-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1756886860253814-13095-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.201.254:35831
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.201.254:35831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:32771/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1756886860253814-13095-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.201.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1756886860253814-13095-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18857
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1756886860253814-13095-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 07:50:08 UTC on 5fd53c4cbb9d
build id 7874
I20250903 08:08:19.766290 18857 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:08:19.767184 18857 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250903 08:08:19.772745 18857 server_base.cc:1047] running on GCE node
W20250903 08:08:19.772831 18865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:19.772874 18862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:19.772843 18863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:08:19.773420 18857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:08:19.774277 18857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:08:19.775470 18857 hybrid_clock.cc:648] HybridClock initialized: now 1756886899775417 us; error 77 us; skew 500 ppm
I20250903 08:08:19.777318 18857 webserver.cc:480] Webserver started at http://127.12.201.254:37535/ using document root <none> and password file <none>
I20250903 08:08:19.777865 18857 fs_manager.cc:362] Metadata directory not provided
I20250903 08:08:19.777932 18857 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:08:19.778111 18857 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:08:19.779847 18857 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1756886860253814-13095-0/minicluster-data/master-0/data/instance:
uuid: "d2bcc63ad9b44b51a20f0f76eae3edec"
format_stamp: "Formatted at 2025-09-03 08:08:19 on dist-test-slave-27r9"
server_key: "1a2910f2c23d18a21ad065c0b0e58016"
server_key_iv: "51d3cb87342063bd1ef33453b6b6d00e"
server_key_version: "encryptionkey@0"
I20250903 08:08:19.780328 18857 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1756886860253814-13095-0/minicluster-data/master-0/wal/instance:
uuid: "d2bcc63ad9b44b51a20f0f76eae3edec"
format_stamp: "Formatted at 2025-09-03 08:08:19 on dist-test-slave-27r9"
server_key: "1a2910f2c23d18a21ad065c0b0e58016"
server_key_iv: "51d3cb87342063bd1ef33453b6b6d00e"
server_key_version: "encryptionkey@0"
I20250903 08:08:19.783524 18857 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20250903 08:08:19.785739 18871 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:19.786964 18857 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250903 08:08:19.787110 18857 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1756886860253814-13095-0/minicluster-data/master-0/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1756886860253814-13095-0/minicluster-data/master-0/wal
uuid: "d2bcc63ad9b44b51a20f0f76eae3edec"
format_stamp: "Formatted at 2025-09-03 08:08:19 on dist-test-slave-27r9"
server_key: "1a2910f2c23d18a21ad065c0b0e58016"
server_key_iv: "51d3cb87342063bd1ef33453b6b6d00e"
server_key_version: "encryptionkey@0"
I20250903 08:08:19.787238 18857 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1756886860253814-13095-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1756886860253814-13095-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1756886860253814-13095-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:08:19.801653 18857 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:08:19.802172 18857 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:08:19.802313 18857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:08:19.809303 18923 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.201.254:35831 every 8 connection(s)
I20250903 08:08:19.809316 18857 rpc_server.cc:307] RPC server started. Bound to: 127.12.201.254:35831
I20250903 08:08:19.810328 18857 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1756886860253814-13095-0/minicluster-data/master-0/data/info.pb
I20250903 08:08:19.812793 18924 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:08:19.815382 13095 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUML90r/build/debug/bin/kudu as pid 18857
I20250903 08:08:19.815481 13095 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1756886860253814-13095-0/minicluster-data/master-0/wal/instance
I20250903 08:08:19.815712 13095 external_mini_cluster.cc:1468] Setting key 30033ad8e817328830fa4fea9acfaa3c
I20250903 08:08:19.818377 18924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2bcc63ad9b44b51a20f0f76eae3edec: Bootstrap starting.
I20250903 08:08:19.820727 18924 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2bcc63ad9b44b51a20f0f76eae3edec: Neither blocks nor log segments found. Creating new log.
I20250903 08:08:19.821532 18924 log.cc:826] T 00000000000000000000000000000000 P d2bcc63ad9b44b51a20f0f76eae3edec: Log is configured to *not* fsync() on all Append() calls
I20250903 08:08:19.823112 18927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:19.816887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33936 (local address 127.12.201.254:35831)
0903 08:08:19.817497 (+ 610us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:19.817506 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:19.817539 (+ 33us) server_negotiation.cc:408] Connection header received
0903 08:08:19.818009 (+ 470us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:19.818024 (+ 15us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:19.818359 (+ 335us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:19.818638 (+ 279us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:19.819182 (+ 544us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:19.819911 (+ 729us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:19.820771 (+ 860us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:19.821043 (+ 272us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:19.821566 (+ 523us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:19.821583 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:19.821593 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0903 08:08:19.821984 (+ 391us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:19.822033 (+ 49us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0903 08:08:19.822052 (+ 19us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:19.822159 (+ 107us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:19.822311 (+ 152us) server_negotiation.cc:300] Negotiation successful
0903 08:08:19.822549 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":125,"threads_started":1}
I20250903 08:08:19.823340 18924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2bcc63ad9b44b51a20f0f76eae3edec: No bootstrap required, opened a new log
I20250903 08:08:19.825659 13095 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20250903 08:08:19.825985 18924 raft_consensus.cc:357] T 00000000000000000000000000000000 P d2bcc63ad9b44b51a20f0f76eae3edec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2bcc63ad9b44b51a20f0f76eae3edec" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 35831 } }
I20250903 08:08:19.826201 18924 raft_consensus.cc:383] T 00000000000000000000000000000000 P d2bcc63ad9b44b51a20f0f76eae3edec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:08:19.826259 18924 raft_consensus.cc:738] T 00000000000000000000000000000000 P d2bcc63ad9b44b51a20f0f76eae3edec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2bcc63ad9b44b51a20f0f76eae3edec, State: Initialized, Role: FOLLOWER
I20250903 08:08:19.826663 18924 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2bcc63ad9b44b51a20f0f76eae3edec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2bcc63ad9b44b51a20f0f76eae3edec" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 35831 } }
I20250903 08:08:19.826789 18924 raft_consensus.cc:397] T 00000000000000000000000000000000 P d2bcc63ad9b44b51a20f0f76eae3edec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250903 08:08:19.826850 18924 raft_consensus.cc:491] T 00000000000000000000000000000000 P d2bcc63ad9b44b51a20f0f76eae3edec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250903 08:08:19.826957 18924 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d2bcc63ad9b44b51a20f0f76eae3edec [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:08:19.827802 18924 raft_consensus.cc:513] T 00000000000000000000000000000000 P d2bcc63ad9b44b51a20f0f76eae3edec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2bcc63ad9b44b51a20f0f76eae3edec" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 35831 } }
I20250903 08:08:19.828078 18924 leader_election.cc:304] T 00000000000000000000000000000000 P d2bcc63ad9b44b51a20f0f76eae3edec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2bcc63ad9b44b51a20f0f76eae3edec; no voters:
I20250903 08:08:19.828325 18924 leader_election.cc:290] T 00000000000000000000000000000000 P d2bcc63ad9b44b51a20f0f76eae3edec [CANDIDATE]: Term 1 election: Requested vote from peers
I20250903 08:08:19.828437 18929 raft_consensus.cc:2802] T 00000000000000000000000000000000 P d2bcc63ad9b44b51a20f0f76eae3edec [term 1 FOLLOWER]: Leader election won for term 1
I20250903 08:08:19.828861 18929 raft_consensus.cc:695] T 00000000000000000000000000000000 P d2bcc63ad9b44b51a20f0f76eae3edec [term 1 LEADER]: Becoming Leader. State: Replica: d2bcc63ad9b44b51a20f0f76eae3edec, State: Running, Role: LEADER
I20250903 08:08:19.829164 18929 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2bcc63ad9b44b51a20f0f76eae3edec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2bcc63ad9b44b51a20f0f76eae3edec" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 35831 } }
I20250903 08:08:19.829371 18924 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2bcc63ad9b44b51a20f0f76eae3edec [sys.catalog]: configured and running, proceeding with master startup.
I20250903 08:08:19.830713 18931 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2bcc63ad9b44b51a20f0f76eae3edec [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2bcc63ad9b44b51a20f0f76eae3edec. Latest consensus state: current_term: 1 leader_uuid: "d2bcc63ad9b44b51a20f0f76eae3edec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2bcc63ad9b44b51a20f0f76eae3edec" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 35831 } } }
I20250903 08:08:19.830829 18931 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2bcc63ad9b44b51a20f0f76eae3edec [sys.catalog]: This master's current role is: LEADER
I20250903 08:08:19.830863 18930 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2bcc63ad9b44b51a20f0f76eae3edec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d2bcc63ad9b44b51a20f0f76eae3edec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2bcc63ad9b44b51a20f0f76eae3edec" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 35831 } } }
I20250903 08:08:19.830964 18930 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2bcc63ad9b44b51a20f0f76eae3edec [sys.catalog]: This master's current role is: LEADER
I20250903 08:08:19.831245 18939 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250903 08:08:19.833778 18939 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250903 08:08:19.838699 18939 catalog_manager.cc:1349] Generated new cluster ID: b775ec2b7e384ce79deefe12f3de5739
I20250903 08:08:19.838759 18939 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250903 08:08:19.852768 18939 catalog_manager.cc:1372] Generated new certificate authority record
I20250903 08:08:19.853892 18939 catalog_manager.cc:1506] Loading token signing keys...
I20250903 08:08:19.861917 18939 catalog_manager.cc:5955] T 00000000000000000000000000000000 P d2bcc63ad9b44b51a20f0f76eae3edec: Generated new TSK 0
I20250903 08:08:19.862718 18939 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250903 08:08:19.897481 18927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:19.873657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33942 (local address 127.12.201.254:35831)
0903 08:08:19.873770 (+ 113us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:19.873774 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:19.873947 (+ 173us) server_negotiation.cc:408] Connection header received
0903 08:08:19.874134 (+ 187us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:19.874138 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:19.874201 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:19.874313 (+ 112us) server_negotiation.cc:227] Negotiated authn=JWT
0903 08:08:19.874665 (+ 352us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:19.875498 (+ 833us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:19.876180 (+ 682us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:19.876381 (+ 201us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:19.876495 (+ 114us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0903 08:08:19.896906 (+ 20411us) server_negotiation.cc:378] Sending JWT_EXCHANGE NegotiatePB response
0903 08:08:19.897044 (+ 138us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:19.897261 (+ 217us) server_negotiation.cc:300] Negotiation successful
0903 08:08:19.897319 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34,"thread_start_us":135,"threads_started":1}
I20250903 08:08:19.909437 18927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:19.905552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33952 (local address 127.12.201.254:35831)
0903 08:08:19.905683 (+ 131us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:19.905687 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:19.905754 (+ 67us) server_negotiation.cc:408] Connection header received
0903 08:08:19.905884 (+ 130us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:19.905888 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:19.905936 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:19.906020 (+ 84us) server_negotiation.cc:227] Negotiated authn=JWT
0903 08:08:19.906321 (+ 301us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:19.907120 (+ 799us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:19.907727 (+ 607us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:19.907921 (+ 194us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:19.907969 (+ 48us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0903 08:08:19.909089 (+ 1120us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
0903 08:08:19.909306 (+ 217us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33952: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
Metrics: {"server-negotiator.queue_time_us":52}
W20250903 08:08:19.909510 18927 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33952: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
W20250903 08:08:19.910122 18960 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 08:08:19.905450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:35831 (local address 127.0.0.1:33952)
0903 08:08:19.905695 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:19.905707 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:19.905791 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:19.906028 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:19.906030 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:19.906036 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0903 08:08:19.906227 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:19.906233 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:19.907238 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:19.907240 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:19.907624 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:19.907632 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:19.907720 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:19.907737 (+ 17us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0903 08:08:19.909272 (+ 1535us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
0903 08:08:19.909398 (+ 126us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.12.201.254:35831: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":84,"threads_started":1}
I20250903 08:08:34.917873 18966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:34.914429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55692 (local address 127.12.201.254:35831)
0903 08:08:34.914706 (+ 277us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:34.914710 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:34.914728 (+ 18us) server_negotiation.cc:408] Connection header received
0903 08:08:34.914882 (+ 154us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:34.914885 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:34.914933 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:34.915028 (+ 95us) server_negotiation.cc:227] Negotiated authn=JWT
0903 08:08:34.915498 (+ 470us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:34.916029 (+ 531us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:34.916683 (+ 654us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:34.916873 (+ 190us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:34.916921 (+ 48us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0903 08:08:34.917535 (+ 614us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
0903 08:08:34.917710 (+ 175us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55692: JWT verification failed: token expired
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":76,"threads_started":1}
W20250903 08:08:34.917932 18965 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 08:08:34.914302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:35831 (local address 127.0.0.1:55692)
0903 08:08:34.914594 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:34.914611 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:34.914728 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:34.915175 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:34.915178 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:34.915185 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0903 08:08:34.915376 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:34.915383 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:34.916163 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:34.916167 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:34.916572 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:34.916580 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:34.916683 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:34.916709 (+ 26us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0903 08:08:34.917660 (+ 951us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
0903 08:08:34.917784 (+ 124us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.12.201.254:35831: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":95,"threads_started":1}
W20250903 08:08:34.917955 18966 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55692: JWT verification failed: token expired
W20250903 08:08:34.920351 18966 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 08:08:34.919658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55694 (local address 127.12.201.254:35831)
0903 08:08:34.919757 (+ 99us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:34.919760 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:34.919806 (+ 46us) server_negotiation.cc:408] Connection header received
0903 08:08:34.919944 (+ 138us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:34.919947 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:34.919996 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:34.920092 (+ 96us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:34.920278 (+ 186us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55694: BlockingRecv error: recv got EOF from 127.0.0.1:55694 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20250903 08:08:34.923780 18976 client_negotiation.cc:375] the client has a JWT but it isn't advertising its JWT-based authn capability since it doesn't trust the certificate of the server at 127.12.201.254:35831
W20250903 08:08:34.924270 18966 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 08:08:34.923541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55704 (local address 127.12.201.254:35831)
0903 08:08:34.923640 (+ 99us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:34.923643 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:34.923737 (+ 94us) server_negotiation.cc:408] Connection header received
0903 08:08:34.923924 (+ 187us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:34.923927 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:34.923972 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:34.924047 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:34.924186 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55704: BlockingRecv error: recv got EOF from 127.0.0.1:55704 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
I20250903 08:08:34.924598 13095 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUML90r/build/debug/bin/kudu with pid 18857
2025-09-03T08:08:34Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniCluster (15324 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20250903 08:08:34.964993 13095 mini_oidc.cc:150] Starting JWKS server
I20250903 08:08:34.965868 13095 webserver.cc:480] Webserver started at https://127.0.0.1:44245/ using document root <none> and password file <none>
I20250903 08:08:34.965955 13095 mini_oidc.cc:186] Starting OIDC Discovery server
I20250903 08:08:34.966164 13095 webserver.cc:480] Webserver started at http://127.0.0.1:39243/ using document root <none> and password file <none>
2025-09-03T08:08:34Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-03T08:08:34Z Disabled control of system clock
I20250903 08:08:34.973623 13095 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUML90r/build/debug/bin/kudu
/tmp/dist-test-taskUML90r/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1756886860253814-13095-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1756886860253814-13095-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1756886860253814-13095-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1756886860253814-13095-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.201.254:41695
--webserver_interface=127.12.201.254
--webserver_port=0
--builtin_ntp_servers=127.12.201.212:32787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.201.254:41695
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44245/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1756886860253814-13095-0/testchainca.cert with env {}
W20250903 08:08:35.076872 18982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:08:35.077091 18982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:08:35.077170 18982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:08:35.080255 18982 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250903 08:08:35.080312 18982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 08:08:35.080343 18982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:08:35.080374 18982 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250903 08:08:35.080412 18982 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20250903 08:08:35.080454 18982 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20250903 08:08:35.080489 18982 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44245/jwks.json
I20250903 08:08:35.084833 18982 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.201.212:32787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1756886860253814-13095-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1756886860253814-13095-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.201.254:41695
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.201.254:41695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44245/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1756886860253814-13095-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.201.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1756886860253814-13095-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18982
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1756886860253814-13095-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 07:50:08 UTC on 5fd53c4cbb9d
build id 7874
I20250903 08:08:35.085872 18982 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:08:35.086704 18982 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:08:35.091852 18990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:35.091876 18987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:35.091953 18988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:08:35.092252 18982 server_base.cc:1047] running on GCE node
I20250903 08:08:35.092692 18982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:08:35.093533 18982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:08:35.094679 18982 hybrid_clock.cc:648] HybridClock initialized: now 1756886915094662 us; error 31 us; skew 500 ppm
I20250903 08:08:35.096388 18982 webserver.cc:480] Webserver started at http://127.12.201.254:36137/ using document root <none> and password file <none>
I20250903 08:08:35.096920 18982 fs_manager.cc:362] Metadata directory not provided
I20250903 08:08:35.096979 18982 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:08:35.097170 18982 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:08:35.098809 18982 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1756886860253814-13095-0/minicluster-data/master-0/data/instance:
uuid: "fdaa6bf3fbf94a8ba9d5ebaf0438cc60"
format_stamp: "Formatted at 2025-09-03 08:08:35 on dist-test-slave-27r9"
server_key: "e3df431c1751ad6281b388b2baff4b1a"
server_key_iv: "e370b5a9c481102eae75d3cf48727849"
server_key_version: "encryptionkey@0"
I20250903 08:08:35.099308 18982 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1756886860253814-13095-0/minicluster-data/master-0/wal/instance:
uuid: "fdaa6bf3fbf94a8ba9d5ebaf0438cc60"
format_stamp: "Formatted at 2025-09-03 08:08:35 on dist-test-slave-27r9"
server_key: "e3df431c1751ad6281b388b2baff4b1a"
server_key_iv: "e370b5a9c481102eae75d3cf48727849"
server_key_version: "encryptionkey@0"
I20250903 08:08:35.102453 18982 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20250903 08:08:35.104578 18996 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:35.105494 18982 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20250903 08:08:35.105597 18982 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1756886860253814-13095-0/minicluster-data/master-0/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1756886860253814-13095-0/minicluster-data/master-0/wal
uuid: "fdaa6bf3fbf94a8ba9d5ebaf0438cc60"
format_stamp: "Formatted at 2025-09-03 08:08:35 on dist-test-slave-27r9"
server_key: "e3df431c1751ad6281b388b2baff4b1a"
server_key_iv: "e370b5a9c481102eae75d3cf48727849"
server_key_version: "encryptionkey@0"
I20250903 08:08:35.105677 18982 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1756886860253814-13095-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1756886860253814-13095-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1756886860253814-13095-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:08:35.120443 18982 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:08:35.121002 18982 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:08:35.121162 18982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:08:35.127800 18982 rpc_server.cc:307] RPC server started. Bound to: 127.12.201.254:41695
I20250903 08:08:35.127851 19048 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.201.254:41695 every 8 connection(s)
I20250903 08:08:35.128775 18982 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1756886860253814-13095-0/minicluster-data/master-0/data/info.pb
I20250903 08:08:35.131290 19049 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:08:35.136437 19049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdaa6bf3fbf94a8ba9d5ebaf0438cc60: Bootstrap starting.
I20250903 08:08:35.138507 13095 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUML90r/build/debug/bin/kudu as pid 18982
I20250903 08:08:35.138602 13095 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1756886860253814-13095-0/minicluster-data/master-0/wal/instance
I20250903 08:08:35.138595 19049 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fdaa6bf3fbf94a8ba9d5ebaf0438cc60: Neither blocks nor log segments found. Creating new log.
I20250903 08:08:35.138811 13095 external_mini_cluster.cc:1468] Setting key c9f569363d7b8748ab99a29890d56130
I20250903 08:08:35.139370 19049 log.cc:826] T 00000000000000000000000000000000 P fdaa6bf3fbf94a8ba9d5ebaf0438cc60: Log is configured to *not* fsync() on all Append() calls
I20250903 08:08:35.141178 19049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdaa6bf3fbf94a8ba9d5ebaf0438cc60: No bootstrap required, opened a new log
I20250903 08:08:35.143913 19049 raft_consensus.cc:357] T 00000000000000000000000000000000 P fdaa6bf3fbf94a8ba9d5ebaf0438cc60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdaa6bf3fbf94a8ba9d5ebaf0438cc60" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 41695 } }
I20250903 08:08:35.144104 19049 raft_consensus.cc:383] T 00000000000000000000000000000000 P fdaa6bf3fbf94a8ba9d5ebaf0438cc60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:08:35.144201 19049 raft_consensus.cc:738] T 00000000000000000000000000000000 P fdaa6bf3fbf94a8ba9d5ebaf0438cc60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdaa6bf3fbf94a8ba9d5ebaf0438cc60, State: Initialized, Role: FOLLOWER
I20250903 08:08:35.144593 19049 consensus_queue.cc:260] T 00000000000000000000000000000000 P fdaa6bf3fbf94a8ba9d5ebaf0438cc60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdaa6bf3fbf94a8ba9d5ebaf0438cc60" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 41695 } }
I20250903 08:08:35.144708 19049 raft_consensus.cc:397] T 00000000000000000000000000000000 P fdaa6bf3fbf94a8ba9d5ebaf0438cc60 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250903 08:08:35.144767 19049 raft_consensus.cc:491] T 00000000000000000000000000000000 P fdaa6bf3fbf94a8ba9d5ebaf0438cc60 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250903 08:08:35.144872 19049 raft_consensus.cc:3058] T 00000000000000000000000000000000 P fdaa6bf3fbf94a8ba9d5ebaf0438cc60 [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:08:35.145659 19049 raft_consensus.cc:513] T 00000000000000000000000000000000 P fdaa6bf3fbf94a8ba9d5ebaf0438cc60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdaa6bf3fbf94a8ba9d5ebaf0438cc60" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 41695 } }
I20250903 08:08:35.145959 19049 leader_election.cc:304] T 00000000000000000000000000000000 P fdaa6bf3fbf94a8ba9d5ebaf0438cc60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fdaa6bf3fbf94a8ba9d5ebaf0438cc60; no voters:
I20250903 08:08:35.146250 19049 leader_election.cc:290] T 00000000000000000000000000000000 P fdaa6bf3fbf94a8ba9d5ebaf0438cc60 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250903 08:08:35.146342 19054 raft_consensus.cc:2802] T 00000000000000000000000000000000 P fdaa6bf3fbf94a8ba9d5ebaf0438cc60 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 08:08:35.146538 19054 raft_consensus.cc:695] T 00000000000000000000000000000000 P fdaa6bf3fbf94a8ba9d5ebaf0438cc60 [term 1 LEADER]: Becoming Leader. State: Replica: fdaa6bf3fbf94a8ba9d5ebaf0438cc60, State: Running, Role: LEADER
I20250903 08:08:35.146868 19054 consensus_queue.cc:237] T 00000000000000000000000000000000 P fdaa6bf3fbf94a8ba9d5ebaf0438cc60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdaa6bf3fbf94a8ba9d5ebaf0438cc60" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 41695 } }
I20250903 08:08:35.147307 19049 sys_catalog.cc:564] T 00000000000000000000000000000000 P fdaa6bf3fbf94a8ba9d5ebaf0438cc60 [sys.catalog]: configured and running, proceeding with master startup.
I20250903 08:08:35.148419 19055 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdaa6bf3fbf94a8ba9d5ebaf0438cc60 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fdaa6bf3fbf94a8ba9d5ebaf0438cc60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdaa6bf3fbf94a8ba9d5ebaf0438cc60" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 41695 } } }
I20250903 08:08:35.148538 19055 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdaa6bf3fbf94a8ba9d5ebaf0438cc60 [sys.catalog]: This master's current role is: LEADER
I20250903 08:08:35.148880 19063 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250903 08:08:35.149792 19056 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdaa6bf3fbf94a8ba9d5ebaf0438cc60 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fdaa6bf3fbf94a8ba9d5ebaf0438cc60. Latest consensus state: current_term: 1 leader_uuid: "fdaa6bf3fbf94a8ba9d5ebaf0438cc60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdaa6bf3fbf94a8ba9d5ebaf0438cc60" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 41695 } } }
I20250903 08:08:35.149871 19056 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdaa6bf3fbf94a8ba9d5ebaf0438cc60 [sys.catalog]: This master's current role is: LEADER
I20250903 08:08:35.151870 19053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:35.139831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44634 (local address 127.12.201.254:41695)
0903 08:08:35.140434 (+ 603us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:35.140443 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:35.140477 (+ 34us) server_negotiation.cc:408] Connection header received
0903 08:08:35.140983 (+ 506us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:35.140999 (+ 16us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:35.141271 (+ 272us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:35.141579 (+ 308us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:35.142100 (+ 521us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:35.142811 (+ 711us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:35.143677 (+ 866us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:35.143869 (+ 192us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:35.149749 (+ 5880us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:35.149772 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:35.149786 (+ 14us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0903 08:08:35.150298 (+ 512us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:35.150351 (+ 53us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0903 08:08:35.150370 (+ 19us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:35.150479 (+ 109us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:35.150876 (+ 397us) server_negotiation.cc:300] Negotiation successful
0903 08:08:35.151155 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":125,"threads_started":1}
I20250903 08:08:35.151993 19063 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250903 08:08:35.154953 13095 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20250903 08:08:35.157232 19063 catalog_manager.cc:1349] Generated new cluster ID: b9fccd819c6543b9badef4f6215d32dd
I20250903 08:08:35.157294 19063 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250903 08:08:35.178222 19063 catalog_manager.cc:1372] Generated new certificate authority record
I20250903 08:08:35.179481 19063 catalog_manager.cc:1506] Loading token signing keys...
I20250903 08:08:35.189822 19063 catalog_manager.cc:5955] T 00000000000000000000000000000000 P fdaa6bf3fbf94a8ba9d5ebaf0438cc60: Generated new TSK 0
I20250903 08:08:35.190654 19063 catalog_manager.cc:1516] Initializing in-progress tserver states...
E20250903 08:08:35.217924 19079 webserver.cc:553] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20250903 08:08:35.218015 19079 webserver.cc:553] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20250903 08:08:35.218562 19053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:35.203028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44650 (local address 127.12.201.254:41695)
0903 08:08:35.203163 (+ 135us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:35.203169 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:35.203262 (+ 93us) server_negotiation.cc:408] Connection header received
0903 08:08:35.203407 (+ 145us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:35.203410 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:35.203460 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:35.203547 (+ 87us) server_negotiation.cc:227] Negotiated authn=JWT
0903 08:08:35.203958 (+ 411us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:35.204469 (+ 511us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:35.205331 (+ 862us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:35.205463 (+ 132us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:35.205542 (+ 79us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0903 08:08:35.218247 (+ 12705us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44245/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0903 08:08:35.218456 (+ 209us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44650: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44245/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":34}
W20250903 08:08:35.218626 19053 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44650: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44245/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20250903 08:08:35.218775 19078 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 08:08:35.202854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:41695 (local address 127.0.0.1:44650)
0903 08:08:35.203162 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:35.203177 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:35.203288 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:35.203552 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:35.203555 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:35.203563 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0903 08:08:35.203854 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:35.203862 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:35.204571 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:35.204576 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:35.205237 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:35.205247 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:35.205353 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:35.205373 (+ 20us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0903 08:08:35.218439 (+ 13066us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44245/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0903 08:08:35.218583 (+ 144us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.12.201.254:41695: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44245/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":157,"threads_started":1}
I20250903 08:08:35.219339 13095 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUML90r/build/debug/bin/kudu with pid 18982
2025-09-03T08:08:35Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (293 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20250903 08:08:35.258625 13095 mini_oidc.cc:150] Starting JWKS server
I20250903 08:08:35.259563 13095 webserver.cc:480] Webserver started at https://127.0.0.1:41491/ using document root <none> and password file <none>
I20250903 08:08:35.259658 13095 mini_oidc.cc:186] Starting OIDC Discovery server
I20250903 08:08:35.259861 13095 webserver.cc:480] Webserver started at http://127.0.0.1:33015/ using document root <none> and password file <none>
2025-09-03T08:08:35Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-03T08:08:35Z Disabled control of system clock
I20250903 08:08:35.267452 13095 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUML90r/build/debug/bin/kudu
/tmp/dist-test-taskUML90r/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1756886860253814-13095-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1756886860253814-13095-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1756886860253814-13095-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1756886860253814-13095-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.201.254:41267
--webserver_interface=127.12.201.254
--webserver_port=0
--builtin_ntp_servers=127.12.201.212:42263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.201.254:41267
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41491/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20250903 08:08:35.374070 19085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:08:35.374337 19085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:08:35.374430 19085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:08:35.377772 19085 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250903 08:08:35.377835 19085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 08:08:35.377863 19085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:08:35.377894 19085 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250903 08:08:35.377934 19085 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20250903 08:08:35.377971 19085 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20250903 08:08:35.378000 19085 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41491/jwks.json
I20250903 08:08:35.381942 19085 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.201.212:42263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1756886860253814-13095-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1756886860253814-13095-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.201.254:41267
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.201.254:41267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41491/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1756886860253814-13095-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.201.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19085
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1756886860253814-13095-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 07:50:08 UTC on 5fd53c4cbb9d
build id 7874
I20250903 08:08:35.383085 19085 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:08:35.384305 19085 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:08:35.389766 19090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:35.389918 19091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:35.390110 19093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:08:35.390496 19085 server_base.cc:1047] running on GCE node
I20250903 08:08:35.391070 19085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:08:35.392112 19085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:08:35.393257 19085 hybrid_clock.cc:648] HybridClock initialized: now 1756886915393245 us; error 45 us; skew 500 ppm
I20250903 08:08:35.394896 19085 webserver.cc:480] Webserver started at http://127.12.201.254:37047/ using document root <none> and password file <none>
I20250903 08:08:35.395469 19085 fs_manager.cc:362] Metadata directory not provided
I20250903 08:08:35.395529 19085 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:08:35.395705 19085 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:08:35.397397 19085 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1756886860253814-13095-0/minicluster-data/master-0/data/instance:
uuid: "a1093d80f2354881992c2e36ebe727e4"
format_stamp: "Formatted at 2025-09-03 08:08:35 on dist-test-slave-27r9"
server_key: "c467b0d52d0d0f1074e8d246427ca69d"
server_key_iv: "7371009dd3833b2ba78acdfa60bdd8ec"
server_key_version: "encryptionkey@0"
I20250903 08:08:35.397854 19085 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1756886860253814-13095-0/minicluster-data/master-0/wal/instance:
uuid: "a1093d80f2354881992c2e36ebe727e4"
format_stamp: "Formatted at 2025-09-03 08:08:35 on dist-test-slave-27r9"
server_key: "c467b0d52d0d0f1074e8d246427ca69d"
server_key_iv: "7371009dd3833b2ba78acdfa60bdd8ec"
server_key_version: "encryptionkey@0"
I20250903 08:08:35.401105 19085 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250903 08:08:35.403311 19099 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:35.404281 19085 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250903 08:08:35.404404 19085 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1756886860253814-13095-0/minicluster-data/master-0/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1756886860253814-13095-0/minicluster-data/master-0/wal
uuid: "a1093d80f2354881992c2e36ebe727e4"
format_stamp: "Formatted at 2025-09-03 08:08:35 on dist-test-slave-27r9"
server_key: "c467b0d52d0d0f1074e8d246427ca69d"
server_key_iv: "7371009dd3833b2ba78acdfa60bdd8ec"
server_key_version: "encryptionkey@0"
I20250903 08:08:35.404495 19085 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1756886860253814-13095-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1756886860253814-13095-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1756886860253814-13095-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:08:35.433756 19085 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:08:35.434444 19085 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:08:35.434621 19085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:08:35.441447 19085 rpc_server.cc:307] RPC server started. Bound to: 127.12.201.254:41267
I20250903 08:08:35.441452 19151 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.201.254:41267 every 8 connection(s)
I20250903 08:08:35.442463 19085 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1756886860253814-13095-0/minicluster-data/master-0/data/info.pb
I20250903 08:08:35.444952 19152 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:08:35.450444 19152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1093d80f2354881992c2e36ebe727e4: Bootstrap starting.
I20250903 08:08:35.452665 19152 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1093d80f2354881992c2e36ebe727e4: Neither blocks nor log segments found. Creating new log.
I20250903 08:08:35.452723 13095 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUML90r/build/debug/bin/kudu as pid 19085
I20250903 08:08:35.452838 13095 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1756886860253814-13095-0/minicluster-data/master-0/wal/instance
I20250903 08:08:35.453042 13095 external_mini_cluster.cc:1468] Setting key ee4d9aff0727253a5ec2f86c68568cb7
I20250903 08:08:35.453401 19152 log.cc:826] T 00000000000000000000000000000000 P a1093d80f2354881992c2e36ebe727e4: Log is configured to *not* fsync() on all Append() calls
I20250903 08:08:35.455358 19152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1093d80f2354881992c2e36ebe727e4: No bootstrap required, opened a new log
I20250903 08:08:35.458181 19152 raft_consensus.cc:357] T 00000000000000000000000000000000 P a1093d80f2354881992c2e36ebe727e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1093d80f2354881992c2e36ebe727e4" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 41267 } }
I20250903 08:08:35.458360 19152 raft_consensus.cc:383] T 00000000000000000000000000000000 P a1093d80f2354881992c2e36ebe727e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:08:35.458424 19152 raft_consensus.cc:738] T 00000000000000000000000000000000 P a1093d80f2354881992c2e36ebe727e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1093d80f2354881992c2e36ebe727e4, State: Initialized, Role: FOLLOWER
I20250903 08:08:35.458860 19152 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1093d80f2354881992c2e36ebe727e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1093d80f2354881992c2e36ebe727e4" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 41267 } }
I20250903 08:08:35.459018 19152 raft_consensus.cc:397] T 00000000000000000000000000000000 P a1093d80f2354881992c2e36ebe727e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250903 08:08:35.459089 19152 raft_consensus.cc:491] T 00000000000000000000000000000000 P a1093d80f2354881992c2e36ebe727e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250903 08:08:35.459208 19152 raft_consensus.cc:3058] T 00000000000000000000000000000000 P a1093d80f2354881992c2e36ebe727e4 [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:08:35.460139 19152 raft_consensus.cc:513] T 00000000000000000000000000000000 P a1093d80f2354881992c2e36ebe727e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1093d80f2354881992c2e36ebe727e4" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 41267 } }
I20250903 08:08:35.460348 19156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:35.453962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48714 (local address 127.12.201.254:41267)
0903 08:08:35.454573 (+ 611us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:35.454587 (+ 14us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:35.454629 (+ 42us) server_negotiation.cc:408] Connection header received
0903 08:08:35.455192 (+ 563us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:35.455211 (+ 19us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:35.455517 (+ 306us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:35.455811 (+ 294us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:35.456351 (+ 540us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:35.457074 (+ 723us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:35.457943 (+ 869us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:35.458173 (+ 230us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:35.458726 (+ 553us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:35.458746 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:35.458757 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0903 08:08:35.459203 (+ 446us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:35.459248 (+ 45us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0903 08:08:35.459264 (+ 16us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:35.459362 (+ 98us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:35.459492 (+ 130us) server_negotiation.cc:300] Negotiation successful
0903 08:08:35.459725 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":135,"threads_started":1}
I20250903 08:08:35.460464 19152 leader_election.cc:304] T 00000000000000000000000000000000 P a1093d80f2354881992c2e36ebe727e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1093d80f2354881992c2e36ebe727e4; no voters:
I20250903 08:08:35.460726 19152 leader_election.cc:290] T 00000000000000000000000000000000 P a1093d80f2354881992c2e36ebe727e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250903 08:08:35.460821 19157 raft_consensus.cc:2802] T 00000000000000000000000000000000 P a1093d80f2354881992c2e36ebe727e4 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 08:08:35.461035 19157 raft_consensus.cc:695] T 00000000000000000000000000000000 P a1093d80f2354881992c2e36ebe727e4 [term 1 LEADER]: Becoming Leader. State: Replica: a1093d80f2354881992c2e36ebe727e4, State: Running, Role: LEADER
I20250903 08:08:35.461337 19157 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1093d80f2354881992c2e36ebe727e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1093d80f2354881992c2e36ebe727e4" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 41267 } }
I20250903 08:08:35.461792 19152 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1093d80f2354881992c2e36ebe727e4 [sys.catalog]: configured and running, proceeding with master startup.
I20250903 08:08:35.462452 19157 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1093d80f2354881992c2e36ebe727e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1093d80f2354881992c2e36ebe727e4. Latest consensus state: current_term: 1 leader_uuid: "a1093d80f2354881992c2e36ebe727e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1093d80f2354881992c2e36ebe727e4" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 41267 } } }
I20250903 08:08:35.462550 19157 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1093d80f2354881992c2e36ebe727e4 [sys.catalog]: This master's current role is: LEADER
I20250903 08:08:35.462839 19158 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1093d80f2354881992c2e36ebe727e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a1093d80f2354881992c2e36ebe727e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1093d80f2354881992c2e36ebe727e4" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 41267 } } }
I20250903 08:08:35.462918 19158 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1093d80f2354881992c2e36ebe727e4 [sys.catalog]: This master's current role is: LEADER
I20250903 08:08:35.463228 19164 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250903 08:08:35.463387 13095 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20250903 08:08:35.466207 19164 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250903 08:08:35.471465 19164 catalog_manager.cc:1349] Generated new cluster ID: 1d9ed4481adf44fb864265e81ae7da5d
I20250903 08:08:35.471527 19164 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250903 08:08:35.490641 19164 catalog_manager.cc:1372] Generated new certificate authority record
I20250903 08:08:35.492007 19164 catalog_manager.cc:1506] Loading token signing keys...
I20250903 08:08:35.502095 19164 catalog_manager.cc:5955] T 00000000000000000000000000000000 P a1093d80f2354881992c2e36ebe727e4: Generated new TSK 0
I20250903 08:08:35.503026 19164 catalog_manager.cc:1516] Initializing in-progress tserver states...
E20250903 08:08:35.539366 19182 webserver.cc:553] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20250903 08:08:35.539451 19182 webserver.cc:553] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20250903 08:08:35.540760 19156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:35.516592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48720 (local address 127.12.201.254:41267)
0903 08:08:35.516716 (+ 124us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:35.516719 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:35.516844 (+ 125us) server_negotiation.cc:408] Connection header received
0903 08:08:35.517013 (+ 169us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:35.517016 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:35.517064 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:35.517144 (+ 80us) server_negotiation.cc:227] Negotiated authn=JWT
0903 08:08:35.517552 (+ 408us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:35.518055 (+ 503us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:35.518906 (+ 851us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:35.519083 (+ 177us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:35.519170 (+ 87us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0903 08:08:35.540349 (+ 21179us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41491/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0903 08:08:35.540647 (+ 298us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48720: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41491/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":52}
W20250903 08:08:35.540819 19156 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48720: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41491/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20250903 08:08:35.540897 19181 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 08:08:35.516448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:41267 (local address 127.0.0.1:48720)
0903 08:08:35.516755 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:35.516770 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:35.516885 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:35.517156 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:35.517160 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:35.517168 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0903 08:08:35.517441 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:35.517450 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:35.518155 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:35.518159 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:35.518804 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:35.518814 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:35.518924 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:35.518945 (+ 21us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0903 08:08:35.540563 (+ 21618us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41491/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0903 08:08:35.540713 (+ 150us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.12.201.254:41267: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41491/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":120,"threads_started":1}
I20250903 08:08:35.541541 13095 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUML90r/build/debug/bin/kudu with pid 19085
2025-09-03T08:08:35Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (322 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2025-09-03T08:08:35Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-03T08:08:35Z Disabled control of system clock
I20250903 08:08:35.588442 13095 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUML90r/build/debug/bin/kudu
/tmp/dist-test-taskUML90r/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1756886860253814-13095-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1756886860253814-13095-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1756886860253814-13095-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1756886860253814-13095-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.201.254:37115
--webserver_interface=127.12.201.254
--webserver_port=0
--builtin_ntp_servers=127.12.201.212:33675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.201.254:37115
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1756886860253814-13095-0/testchainca.cert with env {}
W20250903 08:08:35.692111 19186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:08:35.692353 19186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:08:35.692437 19186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:08:35.695585 19186 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250903 08:08:35.695649 19186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 08:08:35.695676 19186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:08:35.695710 19186 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250903 08:08:35.695749 19186 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20250903 08:08:35.695791 19186 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20250903 08:08:35.695823 19186 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20250903 08:08:35.700030 19186 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.201.212:33675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1756886860253814-13095-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1756886860253814-13095-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.201.254:37115
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.201.254:37115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1756886860253814-13095-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.201.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1756886860253814-13095-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19186
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1756886860253814-13095-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 07:50:08 UTC on 5fd53c4cbb9d
build id 7874
I20250903 08:08:35.701086 19186 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:08:35.701900 19186 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:08:35.707334 19194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:35.707345 19191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:35.707345 19192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:08:35.707523 19186 server_base.cc:1047] running on GCE node
I20250903 08:08:35.708107 19186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:08:35.708940 19186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:08:35.710083 19186 hybrid_clock.cc:648] HybridClock initialized: now 1756886915710064 us; error 34 us; skew 500 ppm
I20250903 08:08:35.711767 19186 webserver.cc:480] Webserver started at http://127.12.201.254:46403/ using document root <none> and password file <none>
I20250903 08:08:35.712340 19186 fs_manager.cc:362] Metadata directory not provided
I20250903 08:08:35.712402 19186 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:08:35.712577 19186 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:08:35.714246 19186 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1756886860253814-13095-0/minicluster-data/master-0/data/instance:
uuid: "14400e2046d54d75893f4cdedf6ce83a"
format_stamp: "Formatted at 2025-09-03 08:08:35 on dist-test-slave-27r9"
server_key: "8690c5b8edf54ed0a7b14d373a779f99"
server_key_iv: "9aa18109f6cf4177e76a9f232a6ae58f"
server_key_version: "encryptionkey@0"
I20250903 08:08:35.714691 19186 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1756886860253814-13095-0/minicluster-data/master-0/wal/instance:
uuid: "14400e2046d54d75893f4cdedf6ce83a"
format_stamp: "Formatted at 2025-09-03 08:08:35 on dist-test-slave-27r9"
server_key: "8690c5b8edf54ed0a7b14d373a779f99"
server_key_iv: "9aa18109f6cf4177e76a9f232a6ae58f"
server_key_version: "encryptionkey@0"
I20250903 08:08:35.717859 19186 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20250903 08:08:35.719986 19200 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:35.720878 19186 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250903 08:08:35.720984 19186 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1756886860253814-13095-0/minicluster-data/master-0/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1756886860253814-13095-0/minicluster-data/master-0/wal
uuid: "14400e2046d54d75893f4cdedf6ce83a"
format_stamp: "Formatted at 2025-09-03 08:08:35 on dist-test-slave-27r9"
server_key: "8690c5b8edf54ed0a7b14d373a779f99"
server_key_iv: "9aa18109f6cf4177e76a9f232a6ae58f"
server_key_version: "encryptionkey@0"
I20250903 08:08:35.721071 19186 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1756886860253814-13095-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1756886860253814-13095-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1756886860253814-13095-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:08:35.745509 19186 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:08:35.746181 19186 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:08:35.746371 19186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:08:35.753602 19252 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.201.254:37115 every 8 connection(s)
I20250903 08:08:35.753605 19186 rpc_server.cc:307] RPC server started. Bound to: 127.12.201.254:37115
I20250903 08:08:35.754624 19186 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1756886860253814-13095-0/minicluster-data/master-0/data/info.pb
I20250903 08:08:35.757175 19253 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:08:35.762638 19253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14400e2046d54d75893f4cdedf6ce83a: Bootstrap starting.
I20250903 08:08:35.763562 13095 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUML90r/build/debug/bin/kudu as pid 19186
I20250903 08:08:35.763684 13095 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1756886860253814-13095-0/minicluster-data/master-0/wal/instance
I20250903 08:08:35.763904 13095 external_mini_cluster.cc:1468] Setting key acbaef92c7df64fa8d9b671d105db5b3
I20250903 08:08:35.765218 19253 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14400e2046d54d75893f4cdedf6ce83a: Neither blocks nor log segments found. Creating new log.
I20250903 08:08:35.766069 19253 log.cc:826] T 00000000000000000000000000000000 P 14400e2046d54d75893f4cdedf6ce83a: Log is configured to *not* fsync() on all Append() calls
I20250903 08:08:35.768213 19253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14400e2046d54d75893f4cdedf6ce83a: No bootstrap required, opened a new log
I20250903 08:08:35.771263 19253 raft_consensus.cc:357] T 00000000000000000000000000000000 P 14400e2046d54d75893f4cdedf6ce83a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14400e2046d54d75893f4cdedf6ce83a" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 37115 } }
I20250903 08:08:35.771487 19253 raft_consensus.cc:383] T 00000000000000000000000000000000 P 14400e2046d54d75893f4cdedf6ce83a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:08:35.771557 19253 raft_consensus.cc:738] T 00000000000000000000000000000000 P 14400e2046d54d75893f4cdedf6ce83a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14400e2046d54d75893f4cdedf6ce83a, State: Initialized, Role: FOLLOWER
I20250903 08:08:35.772043 19253 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14400e2046d54d75893f4cdedf6ce83a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14400e2046d54d75893f4cdedf6ce83a" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 37115 } }
I20250903 08:08:35.772112 19256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:35.764974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57966 (local address 127.12.201.254:37115)
0903 08:08:35.765724 (+ 750us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:35.765736 (+ 12us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:35.765778 (+ 42us) server_negotiation.cc:408] Connection header received
0903 08:08:35.766518 (+ 740us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:35.766539 (+ 21us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:35.766829 (+ 290us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:35.767193 (+ 364us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:35.767670 (+ 477us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:35.768793 (+ 1123us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:35.769481 (+ 688us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:35.769748 (+ 267us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:35.770173 (+ 425us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:35.770198 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:35.770213 (+ 15us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0903 08:08:35.770711 (+ 498us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:35.770764 (+ 53us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0903 08:08:35.770782 (+ 18us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:35.770889 (+ 107us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:35.771083 (+ 194us) server_negotiation.cc:300] Negotiation successful
0903 08:08:35.771330 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":142,"threads_started":1}
I20250903 08:08:35.772205 19253 raft_consensus.cc:397] T 00000000000000000000000000000000 P 14400e2046d54d75893f4cdedf6ce83a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250903 08:08:35.772271 19253 raft_consensus.cc:491] T 00000000000000000000000000000000 P 14400e2046d54d75893f4cdedf6ce83a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250903 08:08:35.772403 19253 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 14400e2046d54d75893f4cdedf6ce83a [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:08:35.773298 19253 raft_consensus.cc:513] T 00000000000000000000000000000000 P 14400e2046d54d75893f4cdedf6ce83a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14400e2046d54d75893f4cdedf6ce83a" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 37115 } }
I20250903 08:08:35.773645 19253 leader_election.cc:304] T 00000000000000000000000000000000 P 14400e2046d54d75893f4cdedf6ce83a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 14400e2046d54d75893f4cdedf6ce83a; no voters:
I20250903 08:08:35.773950 19253 leader_election.cc:290] T 00000000000000000000000000000000 P 14400e2046d54d75893f4cdedf6ce83a [CANDIDATE]: Term 1 election: Requested vote from peers
I20250903 08:08:35.774034 19258 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 14400e2046d54d75893f4cdedf6ce83a [term 1 FOLLOWER]: Leader election won for term 1
I20250903 08:08:35.774209 19258 raft_consensus.cc:695] T 00000000000000000000000000000000 P 14400e2046d54d75893f4cdedf6ce83a [term 1 LEADER]: Becoming Leader. State: Replica: 14400e2046d54d75893f4cdedf6ce83a, State: Running, Role: LEADER
I20250903 08:08:35.774488 19258 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14400e2046d54d75893f4cdedf6ce83a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14400e2046d54d75893f4cdedf6ce83a" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 37115 } }
I20250903 08:08:35.774922 13095 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20250903 08:08:35.775138 19253 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14400e2046d54d75893f4cdedf6ce83a [sys.catalog]: configured and running, proceeding with master startup.
I20250903 08:08:35.775759 19259 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14400e2046d54d75893f4cdedf6ce83a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14400e2046d54d75893f4cdedf6ce83a. Latest consensus state: current_term: 1 leader_uuid: "14400e2046d54d75893f4cdedf6ce83a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14400e2046d54d75893f4cdedf6ce83a" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 37115 } } }
I20250903 08:08:35.775887 19259 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14400e2046d54d75893f4cdedf6ce83a [sys.catalog]: This master's current role is: LEADER
I20250903 08:08:35.776559 19260 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14400e2046d54d75893f4cdedf6ce83a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "14400e2046d54d75893f4cdedf6ce83a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14400e2046d54d75893f4cdedf6ce83a" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 37115 } } }
I20250903 08:08:35.776700 19260 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14400e2046d54d75893f4cdedf6ce83a [sys.catalog]: This master's current role is: LEADER
W20250903 08:08:35.779506 19273 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 14400e2046d54d75893f4cdedf6ce83a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250903 08:08:35.779569 19273 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250903 08:08:35.779639 19264 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250903 08:08:35.780282 19264 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250903 08:08:35.785158 19264 catalog_manager.cc:1349] Generated new cluster ID: 6ea31973f56f4731aa479082e36ef006
I20250903 08:08:35.785226 19264 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250903 08:08:35.820447 19264 catalog_manager.cc:1372] Generated new certificate authority record
I20250903 08:08:35.821503 19264 catalog_manager.cc:1506] Loading token signing keys...
I20250903 08:08:35.829464 19264 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 14400e2046d54d75893f4cdedf6ce83a: Generated new TSK 0
I20250903 08:08:35.830238 19264 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250903 08:08:35.911191 19256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:35.844787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57980 (local address 127.12.201.254:37115)
0903 08:08:35.844912 (+ 125us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:35.844916 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:35.845158 (+ 242us) server_negotiation.cc:408] Connection header received
0903 08:08:35.845313 (+ 155us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:35.845316 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:35.845373 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:35.845462 (+ 89us) server_negotiation.cc:227] Negotiated authn=JWT
0903 08:08:35.845906 (+ 444us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:35.846460 (+ 554us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:35.847394 (+ 934us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:35.847548 (+ 154us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:35.847646 (+ 98us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0903 08:08:35.910674 (+ 63028us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
0903 08:08:35.911015 (+ 341us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57980: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":41}
W20250903 08:08:35.911278 19256 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57980: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
W20250903 08:08:35.911338 19282 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 08:08:35.844631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:37115 (local address 127.0.0.1:57980)
0903 08:08:35.845049 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:35.845064 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:35.845178 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:35.845473 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:35.845476 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:35.845484 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0903 08:08:35.845769 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:35.845778 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:35.846572 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:35.846576 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:35.847263 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:35.847274 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:35.847394 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:35.847419 (+ 25us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0903 08:08:35.910948 (+ 63529us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
0903 08:08:35.911163 (+ 215us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.12.201.254:37115: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":131,"threads_started":1}
I20250903 08:08:35.911984 13095 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUML90r/build/debug/bin/kudu with pid 19186
2025-09-03T08:08:35Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (350 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (115 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (117 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 03 08:08:36 dist-test-slave-27r9 krb5kdc[19294](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 03 08:08:36 dist-test-slave-27r9 krb5kdc[19294](info): set up 2 sockets
Sep 03 08:08:36 dist-test-slave-27r9 krb5kdc[19294](info): commencing operation
krb5kdc: starting...
W20250903 08:08:38.139848 13095 mini_kdc.cc:121] Time spent starting KDC: real 1.975s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/krb5kdc/test-user.keytab.
Sep 03 08:08:38 dist-test-slave-27r9 krb5kdc[19294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-09-03T08:08:38Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-03T08:08:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.201.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.201.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.201.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.201.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.201.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.201.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.201.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.201.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/krb5kdc/kudu.keytab.
I20250903 08:08:38.271571 13095 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUML90r/build/debug/bin/kudu
/tmp/dist-test-taskUML90r/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.201.254:40861
--webserver_interface=127.12.201.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.201.212:33095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.201.254:40861
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 08:08:38.374397 19310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:08:38.374675 19310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:08:38.374770 19310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:08:38.378059 19310 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250903 08:08:38.378140 19310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 08:08:38.378180 19310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:08:38.378212 19310 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250903 08:08:38.378253 19310 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250903 08:08:38.382568 19310 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.201.212:33095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.201.254:40861
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.201.254:40861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.201.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19310
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 07:50:08 UTC on 5fd53c4cbb9d
build id 7874
I20250903 08:08:38.383668 19310 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:08:38.384531 19310 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:08:38.389941 19318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:38.389909 19315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:08:38.390089 19310 server_base.cc:1047] running on GCE node
W20250903 08:08:38.389909 19316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:08:38.390642 19310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:08:38.391587 19310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:08:38.392743 19310 hybrid_clock.cc:648] HybridClock initialized: now 1756886918392728 us; error 43 us; skew 500 ppm
Sep 03 08:08:38 dist-test-slave-27r9 krb5kdc[19294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886918, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 08:08:38.395483 19310 init.cc:377] Logged in from keytab as kudu/127.12.201.254@KRBTEST.COM (short username kudu)
I20250903 08:08:38.396510 19310 webserver.cc:480] Webserver started at http://127.12.201.254:40649/ using document root <none> and password file <none>
I20250903 08:08:38.397044 19310 fs_manager.cc:362] Metadata directory not provided
I20250903 08:08:38.397092 19310 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:08:38.397248 19310 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:08:38.398939 19310 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/minicluster-data/master-0/data/instance:
uuid: "2200f026446b4141935b67729aba85f4"
format_stamp: "Formatted at 2025-09-03 08:08:38 on dist-test-slave-27r9"
server_key: "1e69e28bbc8c4dcf6e39d76b24b5b420"
server_key_iv: "808b6e18c8431e34f088296b97fdcc9e"
server_key_version: "encryptionkey@0"
I20250903 08:08:38.399431 19310 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/minicluster-data/master-0/wal/instance:
uuid: "2200f026446b4141935b67729aba85f4"
format_stamp: "Formatted at 2025-09-03 08:08:38 on dist-test-slave-27r9"
server_key: "1e69e28bbc8c4dcf6e39d76b24b5b420"
server_key_iv: "808b6e18c8431e34f088296b97fdcc9e"
server_key_version: "encryptionkey@0"
I20250903 08:08:38.403609 19310 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20250903 08:08:38.405812 19325 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:38.406913 19310 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250903 08:08:38.407051 19310 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/minicluster-data/master-0/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/minicluster-data/master-0/wal
uuid: "2200f026446b4141935b67729aba85f4"
format_stamp: "Formatted at 2025-09-03 08:08:38 on dist-test-slave-27r9"
server_key: "1e69e28bbc8c4dcf6e39d76b24b5b420"
server_key_iv: "808b6e18c8431e34f088296b97fdcc9e"
server_key_version: "encryptionkey@0"
I20250903 08:08:38.407147 19310 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:08:38.431424 19310 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:08:38.434306 19310 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:08:38.434458 19310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:08:38.441426 19377 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.201.254:40861 every 8 connection(s)
I20250903 08:08:38.441423 19310 rpc_server.cc:307] RPC server started. Bound to: 127.12.201.254:40861
I20250903 08:08:38.442430 19310 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/minicluster-data/master-0/data/info.pb
I20250903 08:08:38.444888 19378 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:08:38.446949 13095 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUML90r/build/debug/bin/kudu as pid 19310
I20250903 08:08:38.447084 13095 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/minicluster-data/master-0/wal/instance
I20250903 08:08:38.447321 13095 external_mini_cluster.cc:1468] Setting key 3443c8a196a667e54413fd410e9f9e0a
I20250903 08:08:38.451256 19378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2200f026446b4141935b67729aba85f4: Bootstrap starting.
I20250903 08:08:38.453619 19378 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2200f026446b4141935b67729aba85f4: Neither blocks nor log segments found. Creating new log.
I20250903 08:08:38.454281 19378 log.cc:826] T 00000000000000000000000000000000 P 2200f026446b4141935b67729aba85f4: Log is configured to *not* fsync() on all Append() calls
Sep 03 08:08:38 dist-test-slave-27r9 krb5kdc[19294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.201.254@KRBTEST.COM
I20250903 08:08:38.456344 19378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2200f026446b4141935b67729aba85f4: No bootstrap required, opened a new log
I20250903 08:08:38.459425 19378 raft_consensus.cc:357] T 00000000000000000000000000000000 P 2200f026446b4141935b67729aba85f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2200f026446b4141935b67729aba85f4" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 40861 } }
I20250903 08:08:38.459636 19378 raft_consensus.cc:383] T 00000000000000000000000000000000 P 2200f026446b4141935b67729aba85f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:08:38.459690 19381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:38.448534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39734 (local address 127.12.201.254:40861)
0903 08:08:38.448940 (+ 406us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:38.448952 (+ 12us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:38.448992 (+ 40us) server_negotiation.cc:408] Connection header received
0903 08:08:38.449591 (+ 599us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:38.449615 (+ 24us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:38.449989 (+ 374us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:38.450330 (+ 341us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:38.451116 (+ 786us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:38.452231 (+ 1115us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:38.452861 (+ 630us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:38.453123 (+ 262us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:38.455226 (+ 2103us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:38.455245 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:38.455258 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:08:38.455295 (+ 37us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:38.457299 (+ 2004us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:38.457649 (+ 350us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:38.457654 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:38.457659 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:38.457771 (+ 112us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:38.457933 (+ 162us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:38.457936 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:38.457938 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:38.458319 (+ 381us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:38.458514 (+ 195us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:38.458689 (+ 175us) server_negotiation.cc:300] Negotiation successful
0903 08:08:38.458955 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":122,"threads_started":1}
I20250903 08:08:38.459697 19378 raft_consensus.cc:738] T 00000000000000000000000000000000 P 2200f026446b4141935b67729aba85f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2200f026446b4141935b67729aba85f4, State: Initialized, Role: FOLLOWER
I20250903 08:08:38.460162 19378 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2200f026446b4141935b67729aba85f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2200f026446b4141935b67729aba85f4" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 40861 } }
I20250903 08:08:38.460307 19378 raft_consensus.cc:397] T 00000000000000000000000000000000 P 2200f026446b4141935b67729aba85f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250903 08:08:38.460364 19378 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2200f026446b4141935b67729aba85f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250903 08:08:38.460474 19378 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 2200f026446b4141935b67729aba85f4 [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:08:38.461248 19378 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2200f026446b4141935b67729aba85f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2200f026446b4141935b67729aba85f4" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 40861 } }
I20250903 08:08:38.461602 19378 leader_election.cc:304] T 00000000000000000000000000000000 P 2200f026446b4141935b67729aba85f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2200f026446b4141935b67729aba85f4; no voters:
I20250903 08:08:38.461887 19378 leader_election.cc:290] T 00000000000000000000000000000000 P 2200f026446b4141935b67729aba85f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250903 08:08:38.461987 19383 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 2200f026446b4141935b67729aba85f4 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 08:08:38.462168 19383 raft_consensus.cc:695] T 00000000000000000000000000000000 P 2200f026446b4141935b67729aba85f4 [term 1 LEADER]: Becoming Leader. State: Replica: 2200f026446b4141935b67729aba85f4, State: Running, Role: LEADER
I20250903 08:08:38.462492 19383 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2200f026446b4141935b67729aba85f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2200f026446b4141935b67729aba85f4" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 40861 } }
I20250903 08:08:38.462872 19378 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2200f026446b4141935b67729aba85f4 [sys.catalog]: configured and running, proceeding with master startup.
I20250903 08:08:38.464313 19384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2200f026446b4141935b67729aba85f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2200f026446b4141935b67729aba85f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2200f026446b4141935b67729aba85f4" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 40861 } } }
I20250903 08:08:38.464433 19384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2200f026446b4141935b67729aba85f4 [sys.catalog]: This master's current role is: LEADER
I20250903 08:08:38.464758 19385 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2200f026446b4141935b67729aba85f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2200f026446b4141935b67729aba85f4. Latest consensus state: current_term: 1 leader_uuid: "2200f026446b4141935b67729aba85f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2200f026446b4141935b67729aba85f4" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 40861 } } }
I20250903 08:08:38.464864 19385 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2200f026446b4141935b67729aba85f4 [sys.catalog]: This master's current role is: LEADER
I20250903 08:08:38.465148 19394 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250903 08:08:38.467684 19394 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250903 08:08:38.472815 19394 catalog_manager.cc:1349] Generated new cluster ID: a7093981df4242d68b37741fc1c7b48f
I20250903 08:08:38.472879 19394 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250903 08:08:38.505648 19394 catalog_manager.cc:1372] Generated new certificate authority record
I20250903 08:08:38.506631 19394 catalog_manager.cc:1506] Loading token signing keys...
I20250903 08:08:38.514909 19394 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 2200f026446b4141935b67729aba85f4: Generated new TSK 0
I20250903 08:08:38.515630 19394 catalog_manager.cc:1516] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.201.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.201.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.201.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.201.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.201.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.201.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.201.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.201.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/krb5kdc/kudu.keytab.
I20250903 08:08:38.573837 13095 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUML90r/build/debug/bin/kudu
/tmp/dist-test-taskUML90r/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.201.193:0
--local_ip_for_outbound_sockets=127.12.201.193
--webserver_interface=127.12.201.193
--webserver_port=0
--tserver_master_addrs=127.12.201.254:40861
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.201.212:33095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 08:08:38.679605 19406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:08:38.679850 19406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:08:38.680037 19406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:08:38.683614 19406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 08:08:38.683701 19406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:08:38.683815 19406 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.201.193
I20250903 08:08:38.688351 19406 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.201.212:33095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.201.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.201.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.12.201.254:40861
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19406
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.201.193
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 07:50:08 UTC on 5fd53c4cbb9d
build id 7874
I20250903 08:08:38.689450 19406 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:08:38.690279 19406 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:08:38.696707 19412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:38.696664 19414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:08:38.696872 19406 server_base.cc:1047] running on GCE node
W20250903 08:08:38.696676 19411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:08:38.697360 19406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:08:38.697870 19406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:08:38.699033 19406 hybrid_clock.cc:648] HybridClock initialized: now 1756886918699012 us; error 32 us; skew 500 ppm
Sep 03 08:08:38 dist-test-slave-27r9 krb5kdc[19294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886918, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 08:08:38.701612 19406 init.cc:377] Logged in from keytab as kudu/127.12.201.193@KRBTEST.COM (short username kudu)
I20250903 08:08:38.702557 19406 webserver.cc:480] Webserver started at http://127.12.201.193:42593/ using document root <none> and password file <none>
I20250903 08:08:38.703127 19406 fs_manager.cc:362] Metadata directory not provided
I20250903 08:08:38.703178 19406 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:08:38.703356 19406 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:08:38.705049 19406 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/minicluster-data/ts-0/data/instance:
uuid: "507b5456e36743adbf9030cf65778619"
format_stamp: "Formatted at 2025-09-03 08:08:38 on dist-test-slave-27r9"
server_key: "69535c3f97eadb2ec6dc347792a2adfa"
server_key_iv: "26738060349c1f9e065c8e5d65c7def4"
server_key_version: "encryptionkey@0"
I20250903 08:08:38.705494 19406 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/minicluster-data/ts-0/wal/instance:
uuid: "507b5456e36743adbf9030cf65778619"
format_stamp: "Formatted at 2025-09-03 08:08:38 on dist-test-slave-27r9"
server_key: "69535c3f97eadb2ec6dc347792a2adfa"
server_key_iv: "26738060349c1f9e065c8e5d65c7def4"
server_key_version: "encryptionkey@0"
I20250903 08:08:38.708778 19406 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250903 08:08:38.710856 19421 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:38.711900 19406 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250903 08:08:38.711999 19406 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/minicluster-data/ts-0/wal
uuid: "507b5456e36743adbf9030cf65778619"
format_stamp: "Formatted at 2025-09-03 08:08:38 on dist-test-slave-27r9"
server_key: "69535c3f97eadb2ec6dc347792a2adfa"
server_key_iv: "26738060349c1f9e065c8e5d65c7def4"
server_key_version: "encryptionkey@0"
I20250903 08:08:38.712085 19406 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:08:38.742594 19406 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:08:38.745879 19406 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:08:38.746065 19406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:08:38.746613 19406 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 08:08:38.747488 19406 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 08:08:38.747536 19406 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:38.747594 19406 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 08:08:38.747623 19406 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:38.756168 19406 rpc_server.cc:307] RPC server started. Bound to: 127.12.201.193:37783
I20250903 08:08:38.756201 19534 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.201.193:37783 every 8 connection(s)
I20250903 08:08:38.757071 19406 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/minicluster-data/ts-0/data/info.pb
I20250903 08:08:38.759784 13095 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUML90r/build/debug/bin/kudu as pid 19406
I20250903 08:08:38.759900 13095 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/minicluster-data/ts-0/wal/instance
I20250903 08:08:38.760129 13095 external_mini_cluster.cc:1468] Setting key 43797615bdc0f104ecf61e5db88887d0
Sep 03 08:08:38 dist-test-slave-27r9 krb5kdc[19294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886918, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.193@KRBTEST.COM for kudu/127.12.201.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.201.194@KRBTEST.COM; defaulting to no policy
I20250903 08:08:38.770612 19381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:38.758748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.193:40337 (local address 127.12.201.254:40861)
0903 08:08:38.758878 (+ 130us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:38.758882 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:38.759566 (+ 684us) server_negotiation.cc:408] Connection header received
0903 08:08:38.760454 (+ 888us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:38.760458 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:38.760516 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:38.760599 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:38.762056 (+ 1457us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:38.762826 (+ 770us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:38.764569 (+ 1743us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:38.764738 (+ 169us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:38.767306 (+ 2568us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:38.767326 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:38.767329 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:08:38.767353 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:38.768800 (+ 1447us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:38.769331 (+ 531us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:38.769333 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:38.769335 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:38.769368 (+ 33us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:38.769683 (+ 315us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:38.769685 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:38.769686 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:38.769825 (+ 139us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:38.769912 (+ 87us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:38.770315 (+ 403us) server_negotiation.cc:300] Negotiation successful
0903 08:08:38.770410 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20250903 08:08:38.771251 19537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:38.759039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:40861 (local address 127.12.201.193:40337)
0903 08:08:38.759423 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:38.759459 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:38.760271 (+ 812us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:38.760778 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:38.760788 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:38.761297 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:08:38.761898 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:38.761911 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:38.762998 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:38.763005 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:38.764437 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:38.764451 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:38.764666 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:38.765315 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:08:38.765353 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:08:38.767137 (+ 1784us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:08:38.768922 (+ 1785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:38.768929 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:38.768946 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:38.769241 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:38.769456 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:38.769461 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:38.769466 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:38.769591 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:38.769915 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:08:38.769920 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:08:38.770123 (+ 203us) client_negotiation.cc:769] Sending connection context
0903 08:08:38.770288 (+ 165us) client_negotiation.cc:240] Negotiation successful
0903 08:08:38.770542 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":89,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.201.194@KRBTEST.COM" created.
I20250903 08:08:38.772457 19535 heartbeater.cc:344] Connected to a master server at 127.12.201.254:40861
I20250903 08:08:38.772735 19535 heartbeater.cc:461] Registering TS with master...
I20250903 08:08:38.773371 19535 heartbeater.cc:507] Master 127.12.201.254:40861 requested a full tablet report, sending...
I20250903 08:08:38.775065 19342 ts_manager.cc:194] Registered new tserver with Master: 507b5456e36743adbf9030cf65778619 (127.12.201.193:37783)
I20250903 08:08:38.776202 19342 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.201.193@KRBTEST.COM'} at 127.12.201.193:40337
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.201.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.201.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.201.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.201.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.201.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.201.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/krb5kdc/kudu.keytab.
I20250903 08:08:38.811170 13095 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUML90r/build/debug/bin/kudu
/tmp/dist-test-taskUML90r/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.201.194:0
--local_ip_for_outbound_sockets=127.12.201.194
--webserver_interface=127.12.201.194
--webserver_port=0
--tserver_master_addrs=127.12.201.254:40861
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.201.212:33095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 08:08:38.917627 19542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:08:38.917845 19542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:08:38.917902 19542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:08:38.921073 19542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 08:08:38.921130 19542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:08:38.921207 19542 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.201.194
I20250903 08:08:38.925452 19542 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.201.212:33095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.201.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.201.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.12.201.254:40861
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19542
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.201.194
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 07:50:08 UTC on 5fd53c4cbb9d
build id 7874
I20250903 08:08:38.926517 19542 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:08:38.927409 19542 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:08:38.933811 19547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:38.933818 19548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:38.933825 19550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:08:38.934124 19542 server_base.cc:1047] running on GCE node
I20250903 08:08:38.934516 19542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:08:38.935104 19542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:08:38.936249 19542 hybrid_clock.cc:648] HybridClock initialized: now 1756886918936238 us; error 33 us; skew 500 ppm
Sep 03 08:08:38 dist-test-slave-27r9 krb5kdc[19294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886918, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 08:08:38.938944 19542 init.cc:377] Logged in from keytab as kudu/127.12.201.194@KRBTEST.COM (short username kudu)
I20250903 08:08:38.939929 19542 webserver.cc:480] Webserver started at http://127.12.201.194:44059/ using document root <none> and password file <none>
I20250903 08:08:38.940451 19542 fs_manager.cc:362] Metadata directory not provided
I20250903 08:08:38.940502 19542 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:08:38.940689 19542 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:08:38.942329 19542 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/minicluster-data/ts-1/data/instance:
uuid: "e17845da37074a59b81addd7166fa599"
format_stamp: "Formatted at 2025-09-03 08:08:38 on dist-test-slave-27r9"
server_key: "bd81894135f58f01070407d3b5334ca7"
server_key_iv: "df1cf391c3cd43f898703acff14efb63"
server_key_version: "encryptionkey@0"
I20250903 08:08:38.942783 19542 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/minicluster-data/ts-1/wal/instance:
uuid: "e17845da37074a59b81addd7166fa599"
format_stamp: "Formatted at 2025-09-03 08:08:38 on dist-test-slave-27r9"
server_key: "bd81894135f58f01070407d3b5334ca7"
server_key_iv: "df1cf391c3cd43f898703acff14efb63"
server_key_version: "encryptionkey@0"
I20250903 08:08:38.945940 19542 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20250903 08:08:38.948324 19557 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:38.949519 19542 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250903 08:08:38.949636 19542 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/minicluster-data/ts-1/wal
uuid: "e17845da37074a59b81addd7166fa599"
format_stamp: "Formatted at 2025-09-03 08:08:38 on dist-test-slave-27r9"
server_key: "bd81894135f58f01070407d3b5334ca7"
server_key_iv: "df1cf391c3cd43f898703acff14efb63"
server_key_version: "encryptionkey@0"
I20250903 08:08:38.949733 19542 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:08:38.962906 19542 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:08:38.966076 19542 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:08:38.966257 19542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:08:38.966775 19542 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 08:08:38.967646 19542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 08:08:38.967693 19542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:38.967739 19542 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 08:08:38.967772 19542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:38.976418 19542 rpc_server.cc:307] RPC server started. Bound to: 127.12.201.194:39837
I20250903 08:08:38.976470 19670 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.201.194:39837 every 8 connection(s)
I20250903 08:08:38.977368 19542 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/minicluster-data/ts-1/data/info.pb
Sep 03 08:08:38 dist-test-slave-27r9 krb5kdc[19294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886918, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.194@KRBTEST.COM for kudu/127.12.201.254@KRBTEST.COM
I20250903 08:08:38.986577 13095 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUML90r/build/debug/bin/kudu as pid 19542
I20250903 08:08:38.986698 13095 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/minicluster-data/ts-1/wal/instance
I20250903 08:08:38.986953 13095 external_mini_cluster.cc:1468] Setting key 97aba36b1fdfa52b2d2e2df99f19668d
I20250903 08:08:38.989658 19381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:38.979047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.194:60349 (local address 127.12.201.254:40861)
0903 08:08:38.979181 (+ 134us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:38.979185 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:38.979853 (+ 668us) server_negotiation.cc:408] Connection header received
0903 08:08:38.980684 (+ 831us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:38.980687 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:38.980768 (+ 81us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:38.980848 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:38.982112 (+ 1264us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:38.982604 (+ 492us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:38.983757 (+ 1153us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:38.983948 (+ 191us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:38.986047 (+ 2099us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:38.986068 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:38.986070 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:08:38.986096 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:38.987774 (+ 1678us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:38.988234 (+ 460us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:38.988238 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:38.988240 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:38.988285 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:38.988540 (+ 255us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:38.988543 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:38.988545 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:38.988777 (+ 232us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:38.988883 (+ 106us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:38.989369 (+ 486us) server_negotiation.cc:300] Negotiation successful
0903 08:08:38.989485 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20250903 08:08:38.990329 19673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:38.979306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:40861 (local address 127.12.201.194:60349)
0903 08:08:38.979722 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:38.979759 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:38.980490 (+ 731us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:38.981015 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:38.981022 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:38.981407 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:08:38.981972 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:38.981985 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:38.982723 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:38.982726 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:38.983631 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:38.983640 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:38.983851 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:38.984472 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:08:38.984496 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:08:38.985901 (+ 1405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:08:38.987892 (+ 1991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:38.987898 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:38.987912 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:38.988140 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:38.988366 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:38.988371 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:38.988374 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:38.988456 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:38.988904 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:08:38.988911 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:08:38.989138 (+ 227us) client_negotiation.cc:769] Sending connection context
0903 08:08:38.989328 (+ 190us) client_negotiation.cc:240] Negotiation successful
0903 08:08:38.989601 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":100,"threads_started":1}
I20250903 08:08:38.991380 19671 heartbeater.cc:344] Connected to a master server at 127.12.201.254:40861
I20250903 08:08:38.991655 19671 heartbeater.cc:461] Registering TS with master...
I20250903 08:08:38.992307 19671 heartbeater.cc:507] Master 127.12.201.254:40861 requested a full tablet report, sending...
I20250903 08:08:38.993402 19342 ts_manager.cc:194] Registered new tserver with Master: e17845da37074a59b81addd7166fa599 (127.12.201.194:39837)
I20250903 08:08:38.993952 19342 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.201.194@KRBTEST.COM'} at 127.12.201.194:60349
WARNING: no policy specified for kudu/127.12.201.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.201.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.201.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.201.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.201.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.201.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.201.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.201.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/krb5kdc/kudu.keytab.
I20250903 08:08:39.034499 13095 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUML90r/build/debug/bin/kudu
/tmp/dist-test-taskUML90r/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.201.195:0
--local_ip_for_outbound_sockets=127.12.201.195
--webserver_interface=127.12.201.195
--webserver_port=0
--tserver_master_addrs=127.12.201.254:40861
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.201.212:33095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 08:08:39.137555 19678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:08:39.137778 19678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:08:39.137841 19678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:08:39.141038 19678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 08:08:39.141098 19678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:08:39.141201 19678 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.201.195
I20250903 08:08:39.145401 19678 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.201.212:33095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.201.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.201.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.12.201.254:40861
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19678
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.201.195
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 07:50:08 UTC on 5fd53c4cbb9d
build id 7874
I20250903 08:08:39.146490 19678 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:08:39.147377 19678 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:08:39.153673 19683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:39.153673 19684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:39.153661 19686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:08:39.153941 19678 server_base.cc:1047] running on GCE node
I20250903 08:08:39.154304 19678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:08:39.154877 19678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:08:39.156018 19678 hybrid_clock.cc:648] HybridClock initialized: now 1756886919156002 us; error 32 us; skew 500 ppm
Sep 03 08:08:39 dist-test-slave-27r9 krb5kdc[19294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886919, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 08:08:39.158738 19678 init.cc:377] Logged in from keytab as kudu/127.12.201.195@KRBTEST.COM (short username kudu)
I20250903 08:08:39.159781 19678 webserver.cc:480] Webserver started at http://127.12.201.195:41555/ using document root <none> and password file <none>
I20250903 08:08:39.160319 19678 fs_manager.cc:362] Metadata directory not provided
I20250903 08:08:39.160369 19678 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:08:39.160558 19678 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:08:39.162281 19678 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/minicluster-data/ts-2/data/instance:
uuid: "2340c5c884d448f58ddf233bfa86a3e2"
format_stamp: "Formatted at 2025-09-03 08:08:39 on dist-test-slave-27r9"
server_key: "5ce5b99774948871904aab3700b8356f"
server_key_iv: "3fc111febec7f127ae3daeb0d68caa38"
server_key_version: "encryptionkey@0"
I20250903 08:08:39.162751 19678 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/minicluster-data/ts-2/wal/instance:
uuid: "2340c5c884d448f58ddf233bfa86a3e2"
format_stamp: "Formatted at 2025-09-03 08:08:39 on dist-test-slave-27r9"
server_key: "5ce5b99774948871904aab3700b8356f"
server_key_iv: "3fc111febec7f127ae3daeb0d68caa38"
server_key_version: "encryptionkey@0"
I20250903 08:08:39.166164 19678 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20250903 08:08:39.168308 19693 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:39.169296 19678 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250903 08:08:39.169410 19678 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/minicluster-data/ts-2/wal
uuid: "2340c5c884d448f58ddf233bfa86a3e2"
format_stamp: "Formatted at 2025-09-03 08:08:39 on dist-test-slave-27r9"
server_key: "5ce5b99774948871904aab3700b8356f"
server_key_iv: "3fc111febec7f127ae3daeb0d68caa38"
server_key_version: "encryptionkey@0"
I20250903 08:08:39.169495 19678 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:08:39.178284 19678 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:08:39.181167 19678 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:08:39.181344 19678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:08:39.181857 19678 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 08:08:39.182667 19678 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 08:08:39.182734 19678 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:39.182791 19678 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 08:08:39.182824 19678 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:39.191888 19678 rpc_server.cc:307] RPC server started. Bound to: 127.12.201.195:44391
I20250903 08:08:39.191926 19806 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.201.195:44391 every 8 connection(s)
I20250903 08:08:39.192831 19678 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/minicluster-data/ts-2/data/info.pb
I20250903 08:08:39.199833 13095 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUML90r/build/debug/bin/kudu as pid 19678
I20250903 08:08:39.199970 13095 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/minicluster-data/ts-2/wal/instance
I20250903 08:08:39.200251 13095 external_mini_cluster.cc:1468] Setting key 76cf93bd5ebea25bba60811d2a921f45
Sep 03 08:08:39 dist-test-slave-27r9 krb5kdc[19294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886919, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.195@KRBTEST.COM for kudu/127.12.201.254@KRBTEST.COM
I20250903 08:08:39.204787 19381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:39.194544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.195:54557 (local address 127.12.201.254:40861)
0903 08:08:39.194671 (+ 127us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:39.194674 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:39.195388 (+ 714us) server_negotiation.cc:408] Connection header received
0903 08:08:39.196291 (+ 903us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:39.196294 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:39.196341 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:39.196415 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:39.197623 (+ 1208us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:39.198126 (+ 503us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:39.199262 (+ 1136us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:39.199399 (+ 137us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:39.201781 (+ 2382us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:39.201795 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:39.201798 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:08:39.201826 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:39.203239 (+ 1413us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:39.203665 (+ 426us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:39.203668 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:39.203677 (+ 9us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:39.203726 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:39.203950 (+ 224us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:39.203952 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:39.203953 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:39.204089 (+ 136us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:39.204159 (+ 70us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:39.204563 (+ 404us) server_negotiation.cc:300] Negotiation successful
0903 08:08:39.204663 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20250903 08:08:39.205403 19809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:39.194815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:40861 (local address 127.12.201.195:54557)
0903 08:08:39.195252 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:39.195287 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:39.196091 (+ 804us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:39.196554 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:39.196562 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:39.196962 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:08:39.197495 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:39.197508 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:39.198216 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:39.198219 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:39.199139 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:39.199152 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:39.199336 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:39.199913 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:08:39.199949 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:08:39.201654 (+ 1705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:08:39.203349 (+ 1695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:39.203354 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:39.203362 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:39.203588 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:39.203796 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:39.203798 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:39.203800 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:39.203873 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:39.204158 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:08:39.204163 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:08:39.204370 (+ 207us) client_negotiation.cc:769] Sending connection context
0903 08:08:39.204534 (+ 164us) client_negotiation.cc:240] Negotiation successful
0903 08:08:39.204742 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":146,"threads_started":1}
I20250903 08:08:39.206408 19807 heartbeater.cc:344] Connected to a master server at 127.12.201.254:40861
I20250903 08:08:39.206635 19807 heartbeater.cc:461] Registering TS with master...
I20250903 08:08:39.207171 19807 heartbeater.cc:507] Master 127.12.201.254:40861 requested a full tablet report, sending...
I20250903 08:08:39.208119 19342 ts_manager.cc:194] Registered new tserver with Master: 2340c5c884d448f58ddf233bfa86a3e2 (127.12.201.195:44391)
I20250903 08:08:39.208642 19342 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.201.195@KRBTEST.COM'} at 127.12.201.195:54557
I20250903 08:08:39.213882 13095 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/krb5kdc/test-admin.keytab.
Sep 03 08:08:39 dist-test-slave-27r9 krb5kdc[19294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 08:08:39.227556 13095 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20250903 08:08:39.230041 19814 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 08:08:39.228851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40814 (local address 127.12.201.193:37783)
0903 08:08:39.229127 (+ 276us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:39.229133 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:39.229158 (+ 25us) server_negotiation.cc:408] Connection header received
0903 08:08:39.229210 (+ 52us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:39.229214 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:39.229376 (+ 162us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:39.229496 (+ 120us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:39.229889 (+ 393us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40814: BlockingRecv error: recv got EOF from 127.0.0.1:40814 (error 108)
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":88,"threads_started":1}
W20250903 08:08:39.230149 13095 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/krb5kdc/krb5cc' of type FILE
Sep 03 08:08:39 dist-test-slave-27r9 krb5kdc[19294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 08:08:39.231154 13095 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 03 08:08:39 dist-test-slave-27r9 krb5kdc[19294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 08:08:39.231976 13095 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 03 08:08:39 dist-test-slave-27r9 krb5kdc[19294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.201.193@KRBTEST.COM
I20250903 08:08:39.240236 19814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:39.232733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40830 (local address 127.12.201.193:37783)
0903 08:08:39.232862 (+ 129us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:39.232866 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:39.232945 (+ 79us) server_negotiation.cc:408] Connection header received
0903 08:08:39.233030 (+ 85us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:39.233033 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:39.233069 (+ 36us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:39.233136 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:39.233684 (+ 548us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:39.234364 (+ 680us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:39.234935 (+ 571us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:39.235138 (+ 203us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:39.237073 (+ 1935us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:39.237095 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:39.237106 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:08:39.237131 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:39.238751 (+ 1620us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:39.239194 (+ 443us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:39.239199 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:39.239201 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:39.239252 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:39.239464 (+ 212us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:39.239467 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:39.239469 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:39.239686 (+ 217us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:39.239857 (+ 171us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:39.239984 (+ 127us) server_negotiation.cc:300] Negotiation successful
0903 08:08:39.240103 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20250903 08:08:39.243707 19814 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 08:08:39.242871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40840 (local address 127.12.201.193:37783)
0903 08:08:39.243018 (+ 147us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:39.243024 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:39.243110 (+ 86us) server_negotiation.cc:408] Connection header received
0903 08:08:39.243191 (+ 81us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:39.243193 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:39.243230 (+ 37us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:39.243296 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:39.243602 (+ 306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40840: BlockingRecv error: recv got EOF from 127.0.0.1:40840 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20250903 08:08:39.243997 13095 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/krb5kdc/krb5cc' of type FILE
Sep 03 08:08:39 dist-test-slave-27r9 krb5kdc[19294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 08:08:39.244915 13095 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 03 08:08:39 dist-test-slave-27r9 krb5kdc[19294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 08:08:39.245995 13095 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 03 08:08:39 dist-test-slave-27r9 krb5kdc[19294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.201.193@KRBTEST.COM
I20250903 08:08:39.253386 19814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:39.246763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40846 (local address 127.12.201.193:37783)
0903 08:08:39.246887 (+ 124us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:39.246890 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:39.246939 (+ 49us) server_negotiation.cc:408] Connection header received
0903 08:08:39.247059 (+ 120us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:39.247062 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:39.247097 (+ 35us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:39.247162 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:39.247676 (+ 514us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:39.248155 (+ 479us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:39.248750 (+ 595us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:39.248873 (+ 123us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:39.250763 (+ 1890us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:39.250781 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:39.250783 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:08:39.250806 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:39.252183 (+ 1377us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:39.252507 (+ 324us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:39.252510 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:39.252511 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:39.252544 (+ 33us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:39.252717 (+ 173us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:39.252719 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:39.252720 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:39.252853 (+ 133us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:39.252968 (+ 115us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:39.253102 (+ 134us) server_negotiation.cc:300] Negotiation successful
0903 08:08:39.253195 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20250903 08:08:39.255869 19814 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 08:08:39.254962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40860 (local address 127.12.201.193:37783)
0903 08:08:39.255107 (+ 145us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:39.255110 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:39.255292 (+ 182us) server_negotiation.cc:408] Connection header received
0903 08:08:39.255365 (+ 73us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:39.255367 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:39.255404 (+ 37us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:39.255461 (+ 57us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:39.255749 (+ 288us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40860: BlockingRecv error: recv got EOF from 127.0.0.1:40860 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20250903 08:08:39.255959 13095 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756886860253814-13095-0/krb5kdc/krb5cc' of type FILE
Sep 03 08:08:39 dist-test-slave-27r9 krb5kdc[19294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 08:08:39.256912 13095 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 03 08:08:39 dist-test-slave-27r9 krb5kdc[19294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 08:08:39.257690 13095 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 03 08:08:39 dist-test-slave-27r9 krb5kdc[19294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.201.193@KRBTEST.COM
I20250903 08:08:39.265872 19814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:39.258535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40874 (local address 127.12.201.193:37783)
0903 08:08:39.258656 (+ 121us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:39.258660 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:39.258724 (+ 64us) server_negotiation.cc:408] Connection header received
0903 08:08:39.258836 (+ 112us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:39.258839 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:39.258878 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:39.258953 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:39.259555 (+ 602us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:39.260009 (+ 454us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:39.260579 (+ 570us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:39.260786 (+ 207us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:39.262682 (+ 1896us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:39.262699 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:39.262701 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:08:39.262726 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:39.264297 (+ 1571us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:39.264664 (+ 367us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:39.264668 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:39.264670 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:39.264708 (+ 38us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:39.265012 (+ 304us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:39.265016 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:39.265018 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:39.265205 (+ 187us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:39.265354 (+ 149us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:39.265579 (+ 225us) server_negotiation.cc:300] Negotiation successful
0903 08:08:39.265701 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20250903 08:08:39.266808 13095 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUML90r/build/debug/bin/kudu with pid 19406
I20250903 08:08:39.273577 13095 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUML90r/build/debug/bin/kudu with pid 19542
I20250903 08:08:39.278865 13095 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUML90r/build/debug/bin/kudu with pid 19678
I20250903 08:08:39.284098 13095 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUML90r/build/debug/bin/kudu with pid 19310
2025-09-03T08:08:39Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3141 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 03 08:08:39 dist-test-slave-27r9 krb5kdc[19827](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 03 08:08:39 dist-test-slave-27r9 krb5kdc[19827](info): set up 2 sockets
Sep 03 08:08:39 dist-test-slave-27r9 krb5kdc[19827](info): commencing operation
krb5kdc: starting...
W20250903 08:08:41.283823 13095 mini_kdc.cc:121] Time spent starting KDC: real 1.978s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/krb5kdc/test-user.keytab.
Sep 03 08:08:41 dist-test-slave-27r9 krb5kdc[19827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-09-03T08:08:41Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-03T08:08:41Z Disabled control of system clock
WARNING: no policy specified for oryx/127.12.201.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.201.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.201.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.12.201.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.12.201.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.201.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.201.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.12.201.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/krb5kdc/oryx.keytab.
I20250903 08:08:41.421907 13095 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUML90r/build/debug/bin/kudu
/tmp/dist-test-taskUML90r/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.201.254:34143
--webserver_interface=127.12.201.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/krb5kdc/oryx.keytab
--principal=oryx/127.12.201.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.201.212:37903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.201.254:34143
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 08:08:41.527572 19843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:08:41.527809 19843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:08:41.527864 19843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:08:41.531024 19843 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250903 08:08:41.531087 19843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 08:08:41.531111 19843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:08:41.531131 19843 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250903 08:08:41.531147 19843 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250903 08:08:41.535354 19843 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.201.212:37903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.201.254:34143
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/krb5kdc/oryx.keytab
--principal=oryx/127.12.201.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.201.254:34143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.201.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19843
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 07:50:08 UTC on 5fd53c4cbb9d
build id 7874
I20250903 08:08:41.536368 19843 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:08:41.537223 19843 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:08:41.542578 19848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:41.542590 19849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:08:41.542678 19843 server_base.cc:1047] running on GCE node
W20250903 08:08:41.542598 19851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:08:41.543322 19843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:08:41.544207 19843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:08:41.545382 19843 hybrid_clock.cc:648] HybridClock initialized: now 1756886921545365 us; error 35 us; skew 500 ppm
Sep 03 08:08:41 dist-test-slave-27r9 krb5kdc[19827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886921, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.201.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 08:08:41.548146 19843 init.cc:377] Logged in from keytab as oryx/127.12.201.254@KRBTEST.COM (short username oryx)
I20250903 08:08:41.549232 19843 webserver.cc:480] Webserver started at http://127.12.201.254:45737/ using document root <none> and password file <none>
I20250903 08:08:41.549780 19843 fs_manager.cc:362] Metadata directory not provided
I20250903 08:08:41.549835 19843 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:08:41.550017 19843 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:08:41.551829 19843 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/master-0/data/instance:
uuid: "ff08db2a004746a6ab7e1d0fa91eed38"
format_stamp: "Formatted at 2025-09-03 08:08:41 on dist-test-slave-27r9"
server_key: "8ec963b0123f27fee4b35c23c8dac249"
server_key_iv: "dda74e5b3a571db92d58f3084e7983a3"
server_key_version: "encryptionkey@0"
I20250903 08:08:41.552318 19843 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/master-0/wal/instance:
uuid: "ff08db2a004746a6ab7e1d0fa91eed38"
format_stamp: "Formatted at 2025-09-03 08:08:41 on dist-test-slave-27r9"
server_key: "8ec963b0123f27fee4b35c23c8dac249"
server_key_iv: "dda74e5b3a571db92d58f3084e7983a3"
server_key_version: "encryptionkey@0"
I20250903 08:08:41.555773 19843 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20250903 08:08:41.557982 19858 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:41.558928 19843 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250903 08:08:41.559085 19843 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/master-0/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/master-0/wal
uuid: "ff08db2a004746a6ab7e1d0fa91eed38"
format_stamp: "Formatted at 2025-09-03 08:08:41 on dist-test-slave-27r9"
server_key: "8ec963b0123f27fee4b35c23c8dac249"
server_key_iv: "dda74e5b3a571db92d58f3084e7983a3"
server_key_version: "encryptionkey@0"
I20250903 08:08:41.559180 19843 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:08:41.574895 19843 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:08:41.577930 19843 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:08:41.578116 19843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:08:41.585417 19910 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.201.254:34143 every 8 connection(s)
I20250903 08:08:41.585427 19843 rpc_server.cc:307] RPC server started. Bound to: 127.12.201.254:34143
I20250903 08:08:41.586462 19843 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/master-0/data/info.pb
I20250903 08:08:41.587591 13095 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUML90r/build/debug/bin/kudu as pid 19843
I20250903 08:08:41.587707 13095 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/master-0/wal/instance
I20250903 08:08:41.587934 13095 external_mini_cluster.cc:1468] Setting key a4e3499a38150dd4ce997609e2f0e863
I20250903 08:08:41.589448 19911 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:08:41.595647 19911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff08db2a004746a6ab7e1d0fa91eed38: Bootstrap starting.
Sep 03 08:08:41 dist-test-slave-27r9 krb5kdc[19827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM
I20250903 08:08:41.598026 19911 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff08db2a004746a6ab7e1d0fa91eed38: Neither blocks nor log segments found. Creating new log.
I20250903 08:08:41.598783 19911 log.cc:826] T 00000000000000000000000000000000 P ff08db2a004746a6ab7e1d0fa91eed38: Log is configured to *not* fsync() on all Append() calls
I20250903 08:08:41.600832 19911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff08db2a004746a6ab7e1d0fa91eed38: No bootstrap required, opened a new log
I20250903 08:08:41.601184 19914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:41.589078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37068 (local address 127.12.201.254:34143)
0903 08:08:41.589465 (+ 387us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:41.589475 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:41.589509 (+ 34us) server_negotiation.cc:408] Connection header received
0903 08:08:41.590582 (+ 1073us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:41.590601 (+ 19us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:41.590893 (+ 292us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:41.591221 (+ 328us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:41.592285 (+ 1064us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:41.593059 (+ 774us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:41.593900 (+ 841us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:41.594107 (+ 207us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:41.596684 (+ 2577us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:41.596726 (+ 42us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:41.596737 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:08:41.596769 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:41.598572 (+ 1803us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:41.599092 (+ 520us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:41.599096 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:41.599101 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:41.599176 (+ 75us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:41.599439 (+ 263us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:41.599443 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:41.599445 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:41.599843 (+ 398us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:41.599989 (+ 146us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:41.600289 (+ 300us) server_negotiation.cc:300] Negotiation successful
0903 08:08:41.600561 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":111,"threads_started":1}
I20250903 08:08:41.603588 19911 raft_consensus.cc:357] T 00000000000000000000000000000000 P ff08db2a004746a6ab7e1d0fa91eed38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff08db2a004746a6ab7e1d0fa91eed38" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 34143 } }
I20250903 08:08:41.603763 19911 raft_consensus.cc:383] T 00000000000000000000000000000000 P ff08db2a004746a6ab7e1d0fa91eed38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:08:41.603814 19911 raft_consensus.cc:738] T 00000000000000000000000000000000 P ff08db2a004746a6ab7e1d0fa91eed38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff08db2a004746a6ab7e1d0fa91eed38, State: Initialized, Role: FOLLOWER
I20250903 08:08:41.604202 19911 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff08db2a004746a6ab7e1d0fa91eed38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff08db2a004746a6ab7e1d0fa91eed38" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 34143 } }
I20250903 08:08:41.604322 19911 raft_consensus.cc:397] T 00000000000000000000000000000000 P ff08db2a004746a6ab7e1d0fa91eed38 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250903 08:08:41.604384 19911 raft_consensus.cc:491] T 00000000000000000000000000000000 P ff08db2a004746a6ab7e1d0fa91eed38 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250903 08:08:41.604519 19911 raft_consensus.cc:3058] T 00000000000000000000000000000000 P ff08db2a004746a6ab7e1d0fa91eed38 [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:08:41.605352 19911 raft_consensus.cc:513] T 00000000000000000000000000000000 P ff08db2a004746a6ab7e1d0fa91eed38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff08db2a004746a6ab7e1d0fa91eed38" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 34143 } }
I20250903 08:08:41.605672 19911 leader_election.cc:304] T 00000000000000000000000000000000 P ff08db2a004746a6ab7e1d0fa91eed38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff08db2a004746a6ab7e1d0fa91eed38; no voters:
I20250903 08:08:41.605963 19911 leader_election.cc:290] T 00000000000000000000000000000000 P ff08db2a004746a6ab7e1d0fa91eed38 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250903 08:08:41.606063 19916 raft_consensus.cc:2802] T 00000000000000000000000000000000 P ff08db2a004746a6ab7e1d0fa91eed38 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 08:08:41.606297 19916 raft_consensus.cc:695] T 00000000000000000000000000000000 P ff08db2a004746a6ab7e1d0fa91eed38 [term 1 LEADER]: Becoming Leader. State: Replica: ff08db2a004746a6ab7e1d0fa91eed38, State: Running, Role: LEADER
I20250903 08:08:41.606601 19916 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff08db2a004746a6ab7e1d0fa91eed38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff08db2a004746a6ab7e1d0fa91eed38" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 34143 } }
I20250903 08:08:41.607159 19911 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff08db2a004746a6ab7e1d0fa91eed38 [sys.catalog]: configured and running, proceeding with master startup.
I20250903 08:08:41.608304 19917 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff08db2a004746a6ab7e1d0fa91eed38 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ff08db2a004746a6ab7e1d0fa91eed38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff08db2a004746a6ab7e1d0fa91eed38" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 34143 } } }
I20250903 08:08:41.608471 19917 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff08db2a004746a6ab7e1d0fa91eed38 [sys.catalog]: This master's current role is: LEADER
I20250903 08:08:41.608902 19925 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250903 08:08:41.608317 19918 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff08db2a004746a6ab7e1d0fa91eed38 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff08db2a004746a6ab7e1d0fa91eed38. Latest consensus state: current_term: 1 leader_uuid: "ff08db2a004746a6ab7e1d0fa91eed38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff08db2a004746a6ab7e1d0fa91eed38" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 34143 } } }
I20250903 08:08:41.609797 19918 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff08db2a004746a6ab7e1d0fa91eed38 [sys.catalog]: This master's current role is: LEADER
I20250903 08:08:41.612012 19925 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250903 08:08:41.617317 19925 catalog_manager.cc:1349] Generated new cluster ID: b7b764f24fd443ee92ab54013d2600ab
I20250903 08:08:41.617390 19925 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250903 08:08:41.631146 19925 catalog_manager.cc:1372] Generated new certificate authority record
I20250903 08:08:41.632421 19925 catalog_manager.cc:1506] Loading token signing keys...
I20250903 08:08:41.642617 19925 catalog_manager.cc:5955] T 00000000000000000000000000000000 P ff08db2a004746a6ab7e1d0fa91eed38: Generated new TSK 0
I20250903 08:08:41.643540 19925 catalog_manager.cc:1516] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.12.201.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.201.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.201.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.12.201.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.12.201.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.201.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.201.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.12.201.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/krb5kdc/oryx.keytab.
I20250903 08:08:41.696326 13095 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUML90r/build/debug/bin/kudu
/tmp/dist-test-taskUML90r/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.201.193:0
--local_ip_for_outbound_sockets=127.12.201.193
--webserver_interface=127.12.201.193
--webserver_port=0
--tserver_master_addrs=127.12.201.254:34143
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/krb5kdc/oryx.keytab
--principal=oryx/127.12.201.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.201.212:37903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 08:08:41.806402 19939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:08:41.806650 19939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:08:41.806712 19939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:08:41.810119 19939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 08:08:41.810190 19939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:08:41.810302 19939 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.201.193
I20250903 08:08:41.814600 19939 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.201.212:37903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/krb5kdc/oryx.keytab
--principal=oryx/127.12.201.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.201.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.201.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.12.201.254:34143
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19939
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.201.193
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 07:50:08 UTC on 5fd53c4cbb9d
build id 7874
I20250903 08:08:41.815737 19939 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:08:41.816610 19939 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:08:41.823266 19947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:41.823225 19944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:41.823237 19945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:08:41.823415 19939 server_base.cc:1047] running on GCE node
I20250903 08:08:41.823823 19939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:08:41.824376 19939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:08:41.825523 19939 hybrid_clock.cc:648] HybridClock initialized: now 1756886921825497 us; error 39 us; skew 500 ppm
Sep 03 08:08:41 dist-test-slave-27r9 krb5kdc[19827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886921, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.201.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 08:08:41.828137 19939 init.cc:377] Logged in from keytab as oryx/127.12.201.193@KRBTEST.COM (short username oryx)
I20250903 08:08:41.829140 19939 webserver.cc:480] Webserver started at http://127.12.201.193:44727/ using document root <none> and password file <none>
I20250903 08:08:41.829685 19939 fs_manager.cc:362] Metadata directory not provided
I20250903 08:08:41.829737 19939 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:08:41.829912 19939 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:08:41.831979 19939 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-0/data/instance:
uuid: "a16339bb0a5e4d3e93e3083c087b5fe7"
format_stamp: "Formatted at 2025-09-03 08:08:41 on dist-test-slave-27r9"
server_key: "0093463df70c666b37d84b3104151f8f"
server_key_iv: "46cf715b43c363fafe47dc28353a4411"
server_key_version: "encryptionkey@0"
I20250903 08:08:41.832450 19939 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-0/wal/instance:
uuid: "a16339bb0a5e4d3e93e3083c087b5fe7"
format_stamp: "Formatted at 2025-09-03 08:08:41 on dist-test-slave-27r9"
server_key: "0093463df70c666b37d84b3104151f8f"
server_key_iv: "46cf715b43c363fafe47dc28353a4411"
server_key_version: "encryptionkey@0"
I20250903 08:08:41.835860 19939 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250903 08:08:41.837972 19954 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:41.838948 19939 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250903 08:08:41.839079 19939 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-0/wal
uuid: "a16339bb0a5e4d3e93e3083c087b5fe7"
format_stamp: "Formatted at 2025-09-03 08:08:41 on dist-test-slave-27r9"
server_key: "0093463df70c666b37d84b3104151f8f"
server_key_iv: "46cf715b43c363fafe47dc28353a4411"
server_key_version: "encryptionkey@0"
I20250903 08:08:41.839174 19939 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:08:41.861073 19939 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:08:41.864117 19939 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:08:41.864315 19939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:08:41.864873 19939 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 08:08:41.865751 19939 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 08:08:41.865803 19939 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:41.865862 19939 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 08:08:41.865895 19939 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:41.874809 19939 rpc_server.cc:307] RPC server started. Bound to: 127.12.201.193:43289
I20250903 08:08:41.874847 20067 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.201.193:43289 every 8 connection(s)
I20250903 08:08:41.875801 19939 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-0/data/info.pb
I20250903 08:08:41.882136 13095 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUML90r/build/debug/bin/kudu as pid 19939
I20250903 08:08:41.882269 13095 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-0/wal/instance
I20250903 08:08:41.882551 13095 external_mini_cluster.cc:1468] Setting key 2ab96c17dd264c411df2611b2e3f35a5
Sep 03 08:08:41 dist-test-slave-27r9 krb5kdc[19827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886921, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.201.193@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM
I20250903 08:08:41.888751 19914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:41.877566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.193:44565 (local address 127.12.201.254:34143)
0903 08:08:41.877688 (+ 122us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:41.877692 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:41.878453 (+ 761us) server_negotiation.cc:408] Connection header received
0903 08:08:41.879312 (+ 859us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:41.879316 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:41.879377 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:41.879464 (+ 87us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:41.881099 (+ 1635us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:41.881610 (+ 511us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:41.882394 (+ 784us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:41.882548 (+ 154us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:41.884712 (+ 2164us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:41.884727 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:41.884730 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:08:41.884753 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:41.886594 (+ 1841us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:41.887107 (+ 513us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:41.887114 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:41.887118 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:41.887175 (+ 57us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:41.887538 (+ 363us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:41.887544 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:41.887548 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:41.887755 (+ 207us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:41.887865 (+ 110us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:41.888345 (+ 480us) server_negotiation.cc:300] Negotiation successful
0903 08:08:41.888499 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20250903 08:08:41.889199 20070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:41.877878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:34143 (local address 127.12.201.193:44565)
0903 08:08:41.878309 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:41.878344 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:41.879108 (+ 764us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:41.879631 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:41.879639 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:41.880022 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:08:41.880946 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:41.880959 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:41.881728 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:41.881731 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:41.882262 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:41.882274 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:41.882487 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:41.883156 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:08:41.883180 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:08:41.884560 (+ 1380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:08:41.886725 (+ 2165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:41.886730 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:41.886740 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:41.887001 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:41.887289 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:41.887296 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:41.887301 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:41.887434 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:41.887852 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:08:41.887857 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:08:41.888082 (+ 225us) client_negotiation.cc:769] Sending connection context
0903 08:08:41.888244 (+ 162us) client_negotiation.cc:240] Negotiation successful
0903 08:08:41.888479 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":127,"threads_started":1}
I20250903 08:08:41.890362 20068 heartbeater.cc:344] Connected to a master server at 127.12.201.254:34143
I20250903 08:08:41.890597 20068 heartbeater.cc:461] Registering TS with master...
I20250903 08:08:41.891160 20068 heartbeater.cc:507] Master 127.12.201.254:34143 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.12.201.194@KRBTEST.COM; defaulting to no policy
I20250903 08:08:41.892593 19875 ts_manager.cc:194] Registered new tserver with Master: a16339bb0a5e4d3e93e3083c087b5fe7 (127.12.201.193:43289)
I20250903 08:08:41.893752 19875 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.201.193@KRBTEST.COM'} at 127.12.201.193:44565
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.201.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.201.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.12.201.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.12.201.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.201.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.201.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.12.201.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/krb5kdc/oryx.keytab.
I20250903 08:08:41.934180 13095 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUML90r/build/debug/bin/kudu
/tmp/dist-test-taskUML90r/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.201.194:0
--local_ip_for_outbound_sockets=127.12.201.194
--webserver_interface=127.12.201.194
--webserver_port=0
--tserver_master_addrs=127.12.201.254:34143
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/krb5kdc/oryx.keytab
--principal=oryx/127.12.201.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.201.212:37903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 08:08:42.040125 20075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:08:42.040347 20075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:08:42.040411 20075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:08:42.043632 20075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 08:08:42.043694 20075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:08:42.043805 20075 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.201.194
I20250903 08:08:42.048086 20075 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.201.212:37903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/krb5kdc/oryx.keytab
--principal=oryx/127.12.201.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.201.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.201.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.12.201.254:34143
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20075
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.201.194
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 07:50:08 UTC on 5fd53c4cbb9d
build id 7874
I20250903 08:08:42.049178 20075 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:08:42.050040 20075 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:08:42.056540 20080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:42.056540 20081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:42.056844 20083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:08:42.057008 20075 server_base.cc:1047] running on GCE node
I20250903 08:08:42.057396 20075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:08:42.057986 20075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:08:42.059140 20075 hybrid_clock.cc:648] HybridClock initialized: now 1756886922059119 us; error 41 us; skew 500 ppm
Sep 03 08:08:42 dist-test-slave-27r9 krb5kdc[19827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886922, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.201.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 08:08:42.062646 20075 init.cc:377] Logged in from keytab as oryx/127.12.201.194@KRBTEST.COM (short username oryx)
I20250903 08:08:42.064188 20075 webserver.cc:480] Webserver started at http://127.12.201.194:37469/ using document root <none> and password file <none>
I20250903 08:08:42.064853 20075 fs_manager.cc:362] Metadata directory not provided
I20250903 08:08:42.064900 20075 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:08:42.065096 20075 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:08:42.067399 20075 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-1/data/instance:
uuid: "2b6d679b81794eac94e52a9454286dcd"
format_stamp: "Formatted at 2025-09-03 08:08:42 on dist-test-slave-27r9"
server_key: "67df2e7f0e479cd54c1ac8a5df30fd44"
server_key_iv: "51463c2674076fa112f7ed5bb9ad5df0"
server_key_version: "encryptionkey@0"
I20250903 08:08:42.067945 20075 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-1/wal/instance:
uuid: "2b6d679b81794eac94e52a9454286dcd"
format_stamp: "Formatted at 2025-09-03 08:08:42 on dist-test-slave-27r9"
server_key: "67df2e7f0e479cd54c1ac8a5df30fd44"
server_key_iv: "51463c2674076fa112f7ed5bb9ad5df0"
server_key_version: "encryptionkey@0"
I20250903 08:08:42.071808 20075 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20250903 08:08:42.074169 20090 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:42.075289 20075 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250903 08:08:42.075412 20075 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-1/wal
uuid: "2b6d679b81794eac94e52a9454286dcd"
format_stamp: "Formatted at 2025-09-03 08:08:42 on dist-test-slave-27r9"
server_key: "67df2e7f0e479cd54c1ac8a5df30fd44"
server_key_iv: "51463c2674076fa112f7ed5bb9ad5df0"
server_key_version: "encryptionkey@0"
I20250903 08:08:42.075506 20075 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:08:42.128149 20075 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:08:42.130923 20075 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:08:42.131135 20075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:08:42.131685 20075 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 08:08:42.132524 20075 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 08:08:42.132573 20075 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:42.132635 20075 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 08:08:42.132663 20075 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:42.141419 20075 rpc_server.cc:307] RPC server started. Bound to: 127.12.201.194:41585
I20250903 08:08:42.141443 20203 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.201.194:41585 every 8 connection(s)
I20250903 08:08:42.142410 20075 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-1/data/info.pb
Sep 03 08:08:42 dist-test-slave-27r9 krb5kdc[19827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886922, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.201.194@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM
I20250903 08:08:42.151243 13095 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUML90r/build/debug/bin/kudu as pid 20075
I20250903 08:08:42.151365 13095 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-1/wal/instance
I20250903 08:08:42.151645 13095 external_mini_cluster.cc:1468] Setting key 4df50455246db6ff6630e28ff51ad76e
I20250903 08:08:42.154814 19914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:42.144278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.194:34335 (local address 127.12.201.254:34143)
0903 08:08:42.144447 (+ 169us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:42.144454 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:42.145097 (+ 643us) server_negotiation.cc:408] Connection header received
0903 08:08:42.145964 (+ 867us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:42.145968 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:42.146017 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:42.146104 (+ 87us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:42.147795 (+ 1691us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:42.148304 (+ 509us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:42.148916 (+ 612us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:42.149093 (+ 177us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:42.151192 (+ 2099us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:42.151217 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:42.151222 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:08:42.151259 (+ 37us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:42.152842 (+ 1583us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:42.153355 (+ 513us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:42.153360 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:42.153362 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:42.153412 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:42.153673 (+ 261us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:42.153679 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:42.153684 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:42.153888 (+ 204us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:42.153994 (+ 106us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:42.154455 (+ 461us) server_negotiation.cc:300] Negotiation successful
0903 08:08:42.154616 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20250903 08:08:42.155638 20206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:42.144535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:34143 (local address 127.12.201.194:34335)
0903 08:08:42.144953 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:42.144991 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:42.145754 (+ 763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:42.146259 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:42.146267 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:42.146657 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:08:42.147635 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:42.147650 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:42.148403 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:42.148406 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:42.148794 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:42.148800 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:42.148958 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:42.149555 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:08:42.149576 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:08:42.151040 (+ 1464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:08:42.152955 (+ 1915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:42.152961 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:42.152972 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:42.153255 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:42.153513 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:42.153515 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:42.153517 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:42.153594 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:42.153978 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:08:42.153982 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:08:42.154201 (+ 219us) client_negotiation.cc:769] Sending connection context
0903 08:08:42.154394 (+ 193us) client_negotiation.cc:240] Negotiation successful
0903 08:08:42.154679 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":116,"threads_started":1}
I20250903 08:08:42.156798 20204 heartbeater.cc:344] Connected to a master server at 127.12.201.254:34143
I20250903 08:08:42.157042 20204 heartbeater.cc:461] Registering TS with master...
I20250903 08:08:42.157572 20204 heartbeater.cc:507] Master 127.12.201.254:34143 requested a full tablet report, sending...
I20250903 08:08:42.158589 19875 ts_manager.cc:194] Registered new tserver with Master: 2b6d679b81794eac94e52a9454286dcd (127.12.201.194:41585)
I20250903 08:08:42.159433 19875 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.201.194@KRBTEST.COM'} at 127.12.201.194:34335
WARNING: no policy specified for oryx/127.12.201.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.201.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.201.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.12.201.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.12.201.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.201.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.201.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.12.201.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/krb5kdc/oryx.keytab.
I20250903 08:08:42.199761 13095 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUML90r/build/debug/bin/kudu
/tmp/dist-test-taskUML90r/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.201.195:0
--local_ip_for_outbound_sockets=127.12.201.195
--webserver_interface=127.12.201.195
--webserver_port=0
--tserver_master_addrs=127.12.201.254:34143
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/krb5kdc/oryx.keytab
--principal=oryx/127.12.201.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.201.212:37903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 08:08:42.305187 20211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:08:42.305392 20211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:08:42.305446 20211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:08:42.308534 20211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 08:08:42.308589 20211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:08:42.308665 20211 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.201.195
I20250903 08:08:42.312636 20211 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.201.212:37903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/krb5kdc/oryx.keytab
--principal=oryx/127.12.201.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.201.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.201.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.12.201.254:34143
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20211
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.201.195
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 07:50:08 UTC on 5fd53c4cbb9d
build id 7874
I20250903 08:08:42.313587 20211 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:08:42.314365 20211 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:08:42.320582 20219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:42.320683 20216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:42.320577 20217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:08:42.320859 20211 server_base.cc:1047] running on GCE node
I20250903 08:08:42.321311 20211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:08:42.321844 20211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:08:42.323017 20211 hybrid_clock.cc:648] HybridClock initialized: now 1756886922323004 us; error 33 us; skew 500 ppm
Sep 03 08:08:42 dist-test-slave-27r9 krb5kdc[19827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886922, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.201.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 08:08:42.325603 20211 init.cc:377] Logged in from keytab as oryx/127.12.201.195@KRBTEST.COM (short username oryx)
I20250903 08:08:42.326558 20211 webserver.cc:480] Webserver started at http://127.12.201.195:46171/ using document root <none> and password file <none>
I20250903 08:08:42.327105 20211 fs_manager.cc:362] Metadata directory not provided
I20250903 08:08:42.327157 20211 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:08:42.327322 20211 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:08:42.329015 20211 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-2/data/instance:
uuid: "509ae480ec1c4aacb55daff15185c729"
format_stamp: "Formatted at 2025-09-03 08:08:42 on dist-test-slave-27r9"
server_key: "d2cf5d273e190c2a2afbae9fe5221c90"
server_key_iv: "f3c4c56b9ebb13f0f92df68e568dcea9"
server_key_version: "encryptionkey@0"
I20250903 08:08:42.329458 20211 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-2/wal/instance:
uuid: "509ae480ec1c4aacb55daff15185c729"
format_stamp: "Formatted at 2025-09-03 08:08:42 on dist-test-slave-27r9"
server_key: "d2cf5d273e190c2a2afbae9fe5221c90"
server_key_iv: "f3c4c56b9ebb13f0f92df68e568dcea9"
server_key_version: "encryptionkey@0"
I20250903 08:08:42.332882 20211 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250903 08:08:42.335075 20226 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:42.336107 20211 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250903 08:08:42.336223 20211 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-2/wal
uuid: "509ae480ec1c4aacb55daff15185c729"
format_stamp: "Formatted at 2025-09-03 08:08:42 on dist-test-slave-27r9"
server_key: "d2cf5d273e190c2a2afbae9fe5221c90"
server_key_iv: "f3c4c56b9ebb13f0f92df68e568dcea9"
server_key_version: "encryptionkey@0"
I20250903 08:08:42.336318 20211 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:08:42.350543 20211 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:08:42.353443 20211 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:08:42.353624 20211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:08:42.354164 20211 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 08:08:42.355067 20211 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 08:08:42.355118 20211 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:42.355157 20211 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 08:08:42.355173 20211 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:42.363934 20211 rpc_server.cc:307] RPC server started. Bound to: 127.12.201.195:35143
I20250903 08:08:42.363962 20339 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.201.195:35143 every 8 connection(s)
I20250903 08:08:42.364959 20211 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-2/data/info.pb
Sep 03 08:08:42 dist-test-slave-27r9 krb5kdc[19827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886922, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.201.195@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM
I20250903 08:08:42.375206 13095 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUML90r/build/debug/bin/kudu as pid 20211
I20250903 08:08:42.375344 13095 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-2/wal/instance
I20250903 08:08:42.375617 13095 external_mini_cluster.cc:1468] Setting key f8e5770d1433260000d184b5cf0836ba
I20250903 08:08:42.376749 19914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:42.366623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.195:45831 (local address 127.12.201.254:34143)
0903 08:08:42.366766 (+ 143us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:42.366771 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:42.367469 (+ 698us) server_negotiation.cc:408] Connection header received
0903 08:08:42.368239 (+ 770us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:42.368242 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:42.368291 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:42.368366 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:42.369944 (+ 1578us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:42.370428 (+ 484us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:42.371034 (+ 606us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:42.371163 (+ 129us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:42.373321 (+ 2158us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:42.373338 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:42.373340 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:08:42.373364 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:42.374740 (+ 1376us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:42.375361 (+ 621us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:42.375364 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:42.375365 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:42.375424 (+ 59us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:42.375736 (+ 312us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:42.375739 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:42.375741 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:42.375881 (+ 140us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:42.375984 (+ 103us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:42.376455 (+ 471us) server_negotiation.cc:300] Negotiation successful
0903 08:08:42.376586 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20250903 08:08:42.377450 20342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:42.366875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:34143 (local address 127.12.201.195:45831)
0903 08:08:42.367319 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:42.367354 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:42.368065 (+ 711us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:42.368520 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:42.368529 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:42.368896 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:08:42.369815 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:42.369829 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:42.370534 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:42.370536 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:42.370919 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:42.370924 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:42.371099 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:42.371656 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:08:42.371678 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:08:42.373159 (+ 1481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:08:42.374854 (+ 1695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:42.374860 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:42.374871 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:42.375227 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:42.375509 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:42.375513 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:42.375515 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:42.375630 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:42.375994 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:08:42.375999 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:08:42.376200 (+ 201us) client_negotiation.cc:769] Sending connection context
0903 08:08:42.376382 (+ 182us) client_negotiation.cc:240] Negotiation successful
0903 08:08:42.376635 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":147,"threads_started":1}
I20250903 08:08:42.378428 20340 heartbeater.cc:344] Connected to a master server at 127.12.201.254:34143
I20250903 08:08:42.378666 20340 heartbeater.cc:461] Registering TS with master...
I20250903 08:08:42.379184 20340 heartbeater.cc:507] Master 127.12.201.254:34143 requested a full tablet report, sending...
I20250903 08:08:42.380185 19875 ts_manager.cc:194] Registered new tserver with Master: 509ae480ec1c4aacb55daff15185c729 (127.12.201.195:35143)
I20250903 08:08:42.380722 19875 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.201.195@KRBTEST.COM'} at 127.12.201.195:45831
I20250903 08:08:42.389331 13095 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.12.201.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.201.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.201.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.12.201.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.12.201.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.201.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.201.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.12.201.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/krb5kdc/oryx.keytab.
I20250903 08:08:42.434144 13095 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUML90r/build/debug/bin/kudu
/tmp/dist-test-taskUML90r/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.201.253:42205
--webserver_interface=127.12.201.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/krb5kdc/oryx.keytab
--principal=oryx/127.12.201.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.201.212:37903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.201.254:34143,127.12.201.253:42205
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 08:08:42.539729 20347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:08:42.539964 20347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:08:42.540047 20347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:08:42.543287 20347 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250903 08:08:42.543354 20347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 08:08:42.543388 20347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:08:42.543421 20347 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250903 08:08:42.543460 20347 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250903 08:08:42.547660 20347 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.201.212:37903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.12.201.254:34143,127.12.201.253:42205
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/krb5kdc/oryx.keytab
--principal=oryx/127.12.201.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.201.253:42205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.12.201.253
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20347
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 07:50:08 UTC on 5fd53c4cbb9d
build id 7874
W20250903 08:08:42.547899 20347 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.12.201.254:34143,127.12.201.253:42205'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20250903 08:08:42.548776 20347 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:08:42.549623 20347 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:08:42.554895 20352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:08:42.554963 20347 server_base.cc:1047] running on GCE node
W20250903 08:08:42.555112 20355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:42.555241 20353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:08:42.555683 20347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:08:42.556535 20347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:08:42.557688 20347 hybrid_clock.cc:648] HybridClock initialized: now 1756886922557676 us; error 26 us; skew 500 ppm
Sep 03 08:08:42 dist-test-slave-27r9 krb5kdc[19827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886922, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.201.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 08:08:42.560395 20347 init.cc:377] Logged in from keytab as oryx/127.12.201.253@KRBTEST.COM (short username oryx)
I20250903 08:08:42.561496 20347 webserver.cc:480] Webserver started at http://127.12.201.253:35009/ using document root <none> and password file <none>
I20250903 08:08:42.562036 20347 fs_manager.cc:362] Metadata directory not provided
I20250903 08:08:42.562081 20347 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:08:42.562234 20347 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:08:42.564050 20347 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/master-1/data/instance:
uuid: "094ea437f04c45fb93ad96244e3ca8a7"
format_stamp: "Formatted at 2025-09-03 08:08:42 on dist-test-slave-27r9"
server_key: "da3ee0d928ee66e2b2bfcabdc086cad2"
server_key_iv: "2a7ee5998c0130360df073d3e56d2429"
server_key_version: "encryptionkey@0"
I20250903 08:08:42.564507 20347 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/master-1/wal/instance:
uuid: "094ea437f04c45fb93ad96244e3ca8a7"
format_stamp: "Formatted at 2025-09-03 08:08:42 on dist-test-slave-27r9"
server_key: "da3ee0d928ee66e2b2bfcabdc086cad2"
server_key_iv: "2a7ee5998c0130360df073d3e56d2429"
server_key_version: "encryptionkey@0"
I20250903 08:08:42.567749 20347 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20250903 08:08:42.569837 20362 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:42.570797 20347 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250903 08:08:42.570899 20347 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/master-1/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/master-1/wal
uuid: "094ea437f04c45fb93ad96244e3ca8a7"
format_stamp: "Formatted at 2025-09-03 08:08:42 on dist-test-slave-27r9"
server_key: "da3ee0d928ee66e2b2bfcabdc086cad2"
server_key_iv: "2a7ee5998c0130360df073d3e56d2429"
server_key_version: "encryptionkey@0"
I20250903 08:08:42.571026 20347 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:08:42.582697 20347 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:08:42.585338 20347 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:08:42.585512 20347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:08:42.593410 20347 rpc_server.cc:307] RPC server started. Bound to: 127.12.201.253:42205
I20250903 08:08:42.593410 20414 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.201.253:42205 every 8 connection(s)
I20250903 08:08:42.594411 20347 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/master-1/data/info.pb
I20250903 08:08:42.596951 20415 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:08:42.599519 13095 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUML90r/build/debug/bin/kudu as pid 20347
I20250903 08:08:42.601063 20415 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.201.254" port: 34143 } has no permanent_uuid. Determining permanent_uuid...
Sep 03 08:08:42 dist-test-slave-27r9 krb5kdc[19827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.201.253@KRBTEST.COM
Sep 03 08:08:42 dist-test-slave-27r9 krb5kdc[19827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886922, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.201.253@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM
I20250903 08:08:42.611269 19914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:42.602564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37074 (local address 127.12.201.254:34143)
0903 08:08:42.602685 (+ 121us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:42.602690 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:42.603017 (+ 327us) server_negotiation.cc:408] Connection header received
0903 08:08:42.603159 (+ 142us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:42.603162 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:42.603213 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:42.603283 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:42.604115 (+ 832us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:42.604600 (+ 485us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:42.605316 (+ 716us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:42.605562 (+ 246us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:42.607947 (+ 2385us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:42.607965 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:42.607967 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:08:42.607991 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:42.609631 (+ 1640us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:42.610073 (+ 442us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:42.610076 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:42.610077 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:42.610115 (+ 38us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:42.610334 (+ 219us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:42.610340 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:42.610342 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:42.610502 (+ 160us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:42.610595 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:42.610952 (+ 357us) server_negotiation.cc:300] Negotiation successful
0903 08:08:42.611097 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20250903 08:08:42.611901 20418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:42.600775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38690 (local address 127.12.201.253:42205)
0903 08:08:42.601147 (+ 372us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:42.601156 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:42.601198 (+ 42us) server_negotiation.cc:408] Connection header received
0903 08:08:42.601718 (+ 520us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:42.601736 (+ 18us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:42.602026 (+ 290us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:42.602292 (+ 266us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:42.603381 (+ 1089us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:42.604103 (+ 722us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:42.604744 (+ 641us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:42.605036 (+ 292us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:42.607385 (+ 2349us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:42.607402 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:42.607410 (+ 8us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:08:42.607444 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:42.609430 (+ 1986us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:42.609852 (+ 422us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:42.609857 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:42.609858 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:42.609921 (+ 63us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:42.610155 (+ 234us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:42.610159 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:42.610161 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:42.610496 (+ 335us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:42.610671 (+ 175us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:42.610895 (+ 224us) server_negotiation.cc:300] Negotiation successful
0903 08:08:42.611181 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":103,"threads_started":1}
I20250903 08:08:42.611909 20419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:42.602492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:34143 (local address 127.0.0.1:37074)
0903 08:08:42.602877 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:42.602908 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:42.603048 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:42.603354 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:42.603361 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:42.603724 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:08:42.603964 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:42.603974 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:42.604709 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:42.604714 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:42.605196 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:42.605208 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:42.605362 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:42.605820 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:08:42.605842 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:08:42.607819 (+ 1977us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:08:42.609728 (+ 1909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:42.609736 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:42.609744 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:42.609980 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:42.610196 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:42.610200 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:42.610202 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:42.610254 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:42.610604 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:08:42.610613 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:08:42.610738 (+ 125us) client_negotiation.cc:769] Sending connection context
0903 08:08:42.610897 (+ 159us) client_negotiation.cc:240] Negotiation successful
0903 08:08:42.611285 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"mutex_wait_us":106,"thread_start_us":149,"threads_started":1}
I20250903 08:08:42.613227 20415 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.201.253" port: 42205 } has no permanent_uuid. Determining permanent_uuid...
Sep 03 08:08:42 dist-test-slave-27r9 krb5kdc[19827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886922, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.201.253@KRBTEST.COM for oryx/127.12.201.253@KRBTEST.COM
I20250903 08:08:42.619728 20419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:42.613708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.253:42205 (local address 127.0.0.1:38692)
0903 08:08:42.613826 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:42.613841 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:42.613910 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:42.614090 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:42.614092 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:42.614219 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:08:42.614393 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:42.614398 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:42.615012 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:42.615018 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:42.615357 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:42.615364 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:42.615449 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:42.615918 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:08:42.615929 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:08:42.617018 (+ 1089us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:08:42.618549 (+ 1531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:42.618552 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:42.618554 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:42.618702 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:42.618898 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:42.618901 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:42.618905 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:42.618943 (+ 38us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:42.619250 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:08:42.619253 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:08:42.619318 (+ 65us) client_negotiation.cc:769] Sending connection context
0903 08:08:42.619424 (+ 106us) client_negotiation.cc:240] Negotiation successful
0903 08:08:42.619553 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20250903 08:08:42.619728 20418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:42.613804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38692 (local address 127.12.201.253:42205)
0903 08:08:42.613907 (+ 103us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:42.613910 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:42.613921 (+ 11us) server_negotiation.cc:408] Connection header received
0903 08:08:42.613981 (+ 60us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:42.613983 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:42.614016 (+ 33us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:42.614072 (+ 56us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:42.614477 (+ 405us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:42.614910 (+ 433us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:42.615448 (+ 538us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:42.615577 (+ 129us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:42.617124 (+ 1547us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:42.617137 (+ 13us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:42.617139 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:08:42.617158 (+ 19us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:42.618452 (+ 1294us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:42.618783 (+ 331us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:42.618787 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:42.618790 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:42.618823 (+ 33us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:42.619030 (+ 207us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:42.619034 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:42.619036 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:42.619159 (+ 123us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:42.619253 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:42.619476 (+ 223us) server_negotiation.cc:300] Negotiation successful
0903 08:08:42.619606 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":43}
I20250903 08:08:42.622295 20415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 094ea437f04c45fb93ad96244e3ca8a7: Bootstrap starting.
I20250903 08:08:42.624696 20415 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 094ea437f04c45fb93ad96244e3ca8a7: Neither blocks nor log segments found. Creating new log.
I20250903 08:08:42.625358 20415 log.cc:826] T 00000000000000000000000000000000 P 094ea437f04c45fb93ad96244e3ca8a7: Log is configured to *not* fsync() on all Append() calls
I20250903 08:08:42.627084 20415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 094ea437f04c45fb93ad96244e3ca8a7: No bootstrap required, opened a new log
I20250903 08:08:42.629413 20415 raft_consensus.cc:357] T 00000000000000000000000000000000 P 094ea437f04c45fb93ad96244e3ca8a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff08db2a004746a6ab7e1d0fa91eed38" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 34143 } } peers { permanent_uuid: "094ea437f04c45fb93ad96244e3ca8a7" member_type: VOTER last_known_addr { host: "127.12.201.253" port: 42205 } }
I20250903 08:08:42.629597 20415 raft_consensus.cc:383] T 00000000000000000000000000000000 P 094ea437f04c45fb93ad96244e3ca8a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:08:42.629673 20415 raft_consensus.cc:738] T 00000000000000000000000000000000 P 094ea437f04c45fb93ad96244e3ca8a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 094ea437f04c45fb93ad96244e3ca8a7, State: Initialized, Role: FOLLOWER
I20250903 08:08:42.630084 20415 consensus_queue.cc:260] T 00000000000000000000000000000000 P 094ea437f04c45fb93ad96244e3ca8a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff08db2a004746a6ab7e1d0fa91eed38" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 34143 } } peers { permanent_uuid: "094ea437f04c45fb93ad96244e3ca8a7" member_type: VOTER last_known_addr { host: "127.12.201.253" port: 42205 } }
I20250903 08:08:42.630549 20421 sys_catalog.cc:455] T 00000000000000000000000000000000 P 094ea437f04c45fb93ad96244e3ca8a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff08db2a004746a6ab7e1d0fa91eed38" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 34143 } } peers { permanent_uuid: "094ea437f04c45fb93ad96244e3ca8a7" member_type: VOTER last_known_addr { host: "127.12.201.253" port: 42205 } } }
I20250903 08:08:42.630681 20421 sys_catalog.cc:458] T 00000000000000000000000000000000 P 094ea437f04c45fb93ad96244e3ca8a7 [sys.catalog]: This master's current role is: FOLLOWER
I20250903 08:08:42.631218 20415 sys_catalog.cc:564] T 00000000000000000000000000000000 P 094ea437f04c45fb93ad96244e3ca8a7 [sys.catalog]: configured and running, proceeding with master startup.
W20250903 08:08:42.635741 20432 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 094ea437f04c45fb93ad96244e3ca8a7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250903 08:08:42.635830 20432 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250903 08:08:42.637488 20347 master_runner.cc:428] Detected that this master 094ea437f04c45fb93ad96244e3ca8a7 is joining an existing cluster
I20250903 08:08:42.637544 20347 master_runner.cc:432] Initiating AddMaster RPC to add 127.12.201.253:42205
I20250903 08:08:42.647362 20439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:42.639383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:34143 (local address 127.0.0.1:37082)
0903 08:08:42.639831 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:42.639844 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:42.639928 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:42.640286 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:42.640290 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:42.640557 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:08:42.641273 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:42.641284 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:42.642022 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:42.642026 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:42.642659 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:42.642670 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:42.642807 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:42.643379 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:08:42.643395 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:08:42.643825 (+ 430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:08:42.645902 (+ 2077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:42.645908 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:42.645913 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:42.646106 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:42.646362 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:42.646366 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:42.646371 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:42.646426 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:42.646820 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:08:42.646826 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:08:42.646916 (+ 90us) client_negotiation.cc:769] Sending connection context
0903 08:08:42.647052 (+ 136us) client_negotiation.cc:240] Negotiation successful
0903 08:08:42.647169 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":107,"threads_started":1}
I20250903 08:08:42.647353 19914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:42.639460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37082 (local address 127.12.201.254:34143)
0903 08:08:42.639729 (+ 269us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:42.639733 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:42.639920 (+ 187us) server_negotiation.cc:408] Connection header received
0903 08:08:42.640000 (+ 80us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:42.640004 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:42.640061 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:42.640164 (+ 103us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:42.641401 (+ 1237us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:42.641902 (+ 501us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:42.642803 (+ 901us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:42.643048 (+ 245us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:42.643957 (+ 909us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:42.643981 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:42.643986 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:08:42.644016 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:42.645785 (+ 1769us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:42.646200 (+ 415us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:42.646204 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:42.646205 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:42.646256 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:42.646530 (+ 274us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:42.646534 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:42.646536 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:42.646716 (+ 180us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:42.646827 (+ 111us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:42.647106 (+ 279us) server_negotiation.cc:300] Negotiation successful
0903 08:08:42.647218 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176}
I20250903 08:08:42.651021 20438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:42.639597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.253:42205 (local address 127.0.0.1:38696)
0903 08:08:42.643279 (+ 3682us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:42.643290 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:42.643420 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:42.643659 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:42.643662 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:42.643873 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:08:42.644146 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:42.644156 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:42.645043 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:42.645047 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:42.645644 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:42.645657 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:42.645777 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:42.646264 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:08:42.646280 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:08:42.646628 (+ 348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:08:42.649026 (+ 2398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:42.649030 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:42.649032 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:42.649233 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:42.649630 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:42.649633 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:42.649634 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:42.649689 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:42.650439 (+ 750us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:08:42.650442 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:08:42.650522 (+ 80us) client_negotiation.cc:769] Sending connection context
0903 08:08:42.650643 (+ 121us) client_negotiation.cc:240] Negotiation successful
0903 08:08:42.650772 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3626,"thread_start_us":89,"threads_started":1}
I20250903 08:08:42.651026 20418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:42.639436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38696 (local address 127.12.201.253:42205)
0903 08:08:42.639548 (+ 112us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:42.639551 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:42.643369 (+ 3818us) server_negotiation.cc:408] Connection header received
0903 08:08:42.643538 (+ 169us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:42.643540 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:42.643572 (+ 32us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:42.643639 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:42.644308 (+ 669us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:42.644919 (+ 611us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:42.645778 (+ 859us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:42.646304 (+ 526us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:42.646723 (+ 419us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:42.646737 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:42.646740 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:08:42.646760 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:42.648912 (+ 2152us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:42.649349 (+ 437us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:42.649352 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:42.649354 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:42.649392 (+ 38us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:42.649777 (+ 385us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:42.649782 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:42.649787 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:42.649953 (+ 166us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:42.650055 (+ 102us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:42.650705 (+ 650us) server_negotiation.cc:300] Negotiation successful
0903 08:08:42.650845 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":40}
Sep 03 08:08:42 dist-test-slave-27r9 krb5kdc[19827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886921, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.201.254@KRBTEST.COM for oryx/127.12.201.253@KRBTEST.COM
WARNING: no policy specified for oryx/127.12.201.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.201.252@KRBTEST.COM" created.
I20250903 08:08:42.679308 20418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:42.652421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38708 (local address 127.12.201.253:42205)
0903 08:08:42.652522 (+ 101us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:42.652525 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:42.652535 (+ 10us) server_negotiation.cc:408] Connection header received
0903 08:08:42.652586 (+ 51us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:42.652588 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:42.652625 (+ 37us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:42.652696 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:42.653433 (+ 737us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:42.653833 (+ 400us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:42.654377 (+ 544us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:42.654553 (+ 176us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:42.656544 (+ 1991us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:42.656560 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:42.656563 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:08:42.656589 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:42.677118 (+ 20529us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:42.677884 (+ 766us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:42.677888 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:42.677890 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:42.677944 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:42.678230 (+ 286us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:42.678233 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:42.678235 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:42.678431 (+ 196us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:42.678525 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:42.679002 (+ 477us) server_negotiation.cc:300] Negotiation successful
0903 08:08:42.679127 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20250903 08:08:42.679394 20440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:42.651801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.253:42205 (local address 127.0.0.1:38708)
0903 08:08:42.652103 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:42.652141 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:42.652279 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:42.652754 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:42.652761 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:42.653103 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:08:42.653320 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:42.653326 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:42.653936 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:42.653939 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:42.654280 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:42.654286 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:42.654387 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:42.654948 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:08:42.654960 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:08:42.656411 (+ 1451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:08:42.677313 (+ 20902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:42.677330 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:42.677338 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:42.677758 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:42.678043 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:42.678048 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:42.678052 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:42.678127 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:42.678550 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:08:42.678562 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:08:42.678726 (+ 164us) client_negotiation.cc:769] Sending connection context
0903 08:08:42.678914 (+ 188us) client_negotiation.cc:240] Negotiation successful
0903 08:08:42.679156 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":72,"threads_started":1}
I20250903 08:08:42.681247 19875 catalog_manager.cc:7069] Initiating ChangeConfig request to add master 127.12.201.253:42205: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "094ea437f04c45fb93ad96244e3ca8a7"
member_type: NON_VOTER
last_known_addr {
host: "127.12.201.253"
port: 42205
}
attrs {
promote: true
}
}
dest_uuid: "ff08db2a004746a6ab7e1d0fa91eed38"
cas_config_opid_index: -1
I20250903 08:08:42.681939 19875 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff08db2a004746a6ab7e1d0fa91eed38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ff08db2a004746a6ab7e1d0fa91eed38" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 34143 } } peers { permanent_uuid: "094ea437f04c45fb93ad96244e3ca8a7" member_type: NON_VOTER last_known_addr { host: "127.12.201.253" port: 42205 } attrs { promote: true } }
I20250903 08:08:42.683861 20444 raft_consensus.cc:2953] T 00000000000000000000000000000000 P ff08db2a004746a6ab7e1d0fa91eed38 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 094ea437f04c45fb93ad96244e3ca8a7 (127.12.201.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ff08db2a004746a6ab7e1d0fa91eed38" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 34143 } } peers { permanent_uuid: "094ea437f04c45fb93ad96244e3ca8a7" member_type: NON_VOTER last_known_addr { host: "127.12.201.253" port: 42205 } attrs { promote: true } } }
I20250903 08:08:42.684301 20389 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 094ea437f04c45fb93ad96244e3ca8a7 [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:08:42.685514 20389 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 094ea437f04c45fb93ad96244e3ca8a7 [term 1 FOLLOWER]: Refusing update from remote peer ff08db2a004746a6ab7e1d0fa91eed38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250903 08:08:42.685892 20444 catalog_manager.cc:7073] Successfully completed master ChangeConfig request to add master 127.12.201.253:42205
I20250903 08:08:42.686259 20444 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ff08db2a004746a6ab7e1d0fa91eed38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "094ea437f04c45fb93ad96244e3ca8a7" member_type: NON_VOTER last_known_addr { host: "127.12.201.253" port: 42205 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 08:08:42.686408 20347 master.cc:561] Master@127.12.201.253:42205 shutting down...
I20250903 08:08:42.687536 20445 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff08db2a004746a6ab7e1d0fa91eed38 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ff08db2a004746a6ab7e1d0fa91eed38" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ff08db2a004746a6ab7e1d0fa91eed38" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 34143 } } peers { permanent_uuid: "094ea437f04c45fb93ad96244e3ca8a7" member_type: NON_VOTER last_known_addr { host: "127.12.201.253" port: 42205 } attrs { promote: true } } }
I20250903 08:08:42.687976 20445 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff08db2a004746a6ab7e1d0fa91eed38 [sys.catalog]: This master's current role is: LEADER
W20250903 08:08:42.689237 19862 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.201.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.12.201.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/krb5kdc/oryx.keytab.
W20250903 08:08:42.690737 19862 consensus_peers.cc:489] T 00000000000000000000000000000000 P ff08db2a004746a6ab7e1d0fa91eed38 -> Peer 094ea437f04c45fb93ad96244e3ca8a7 (127.12.201.253:42205): Couldn't send request to peer 094ea437f04c45fb93ad96244e3ca8a7. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20250903 08:08:42.691045 20347 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 094ea437f04c45fb93ad96244e3ca8a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 08:08:42.691298 20347 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 094ea437f04c45fb93ad96244e3ca8a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 08:08:42.691395 20347 tablet_replica.cc:331] T 00000000000000000000000000000000 P 094ea437f04c45fb93ad96244e3ca8a7: stopping tablet replica
I20250903 08:08:42.695578 20347 master.cc:583] Master@127.12.201.253:42205 shutdown complete.
I20250903 08:08:42.695672 20347 master_runner.cc:305] Clearing existing system tablet
I20250903 08:08:42.695942 20347 fs_manager.cc:362] Metadata directory not provided
I20250903 08:08:42.696035 20347 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:08:42.697366 20347 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
WARNING: no policy specified for HTTP/127.12.201.252@KRBTEST.COM; defaulting to no policy
I20250903 08:08:42.698308 20451 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:42.698535 20347 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:08:42.698657 20347 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250903 08:08:42.698693 20347 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/master-1/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/master-1/wal
uuid: "094ea437f04c45fb93ad96244e3ca8a7"
format_stamp: "Formatted at 2025-09-03 08:08:42 on dist-test-slave-27r9"
server_key: "da3ee0d928ee66e2b2bfcabdc086cad2"
server_key_iv: "2a7ee5998c0130360df073d3e56d2429"
server_key_version: "encryptionkey@0"
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.201.252@KRBTEST.COM" created.
I20250903 08:08:42.701058 20347 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 094ea437f04c45fb93ad96244e3ca8a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 08:08:42.703361 20347 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 094ea437f04c45fb93ad96244e3ca8a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20250903 08:08:42.703454 20347 log.cc:1199] T 00000000000000000000000000000000 P 094ea437f04c45fb93ad96244e3ca8a7: Deleting WAL directory at /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20250903 08:08:42.703722 20347 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 094ea437f04c45fb93ad96244e3ca8a7: Deleting consensus metadata
I20250903 08:08:42.703934 20347 master_runner.cc:315] Copying system tablet from 127.12.201.254:34143
I20250903 08:08:42.705159 20347 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P 094ea437f04c45fb93ad96244e3ca8a7: tablet copy: Beginning tablet copy session from remote peer at address 127.12.201.254:34143
I20250903 08:08:42.713812 20458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:42.705814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:34143 (local address 127.0.0.1:37096)
0903 08:08:42.706233 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:42.706251 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:42.706386 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:42.706685 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:42.706691 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:42.706930 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:08:42.707265 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:42.707275 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:42.708285 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:42.708290 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:42.708938 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:42.708949 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:42.709072 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:42.709700 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:08:42.709724 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:08:42.710101 (+ 377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:08:42.712239 (+ 2138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:42.712246 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:42.712253 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:42.712460 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:42.712695 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:42.712697 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:42.712699 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:42.712745 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:42.713220 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:08:42.713223 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:08:42.713309 (+ 86us) client_negotiation.cc:769] Sending connection context
0903 08:08:42.713445 (+ 136us) client_negotiation.cc:240] Negotiation successful
0903 08:08:42.713603 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":104,"threads_started":1}
I20250903 08:08:42.713824 19914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:42.705895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37096 (local address 127.12.201.254:34143)
0903 08:08:42.706030 (+ 135us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:42.706035 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:42.706333 (+ 298us) server_negotiation.cc:408] Connection header received
0903 08:08:42.706491 (+ 158us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:42.706495 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:42.706547 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:42.706647 (+ 100us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:42.707407 (+ 760us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:42.708143 (+ 736us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:42.709054 (+ 911us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:42.709253 (+ 199us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:42.710219 (+ 966us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:42.710240 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:42.710244 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:08:42.710271 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:42.712107 (+ 1836us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:42.712556 (+ 449us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:42.712560 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:42.712562 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:42.712612 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:42.712906 (+ 294us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:42.712910 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:42.712911 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:42.713086 (+ 175us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:42.713207 (+ 121us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:42.713559 (+ 352us) server_negotiation.cc:300] Negotiation successful
0903 08:08:42.713678 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20250903 08:08:42.714756 19895 tablet_copy_service.cc:140] P ff08db2a004746a6ab7e1d0fa91eed38: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 094ea437f04c45fb93ad96244e3ca8a7 ({username='oryx', principal='oryx/127.12.201.253@KRBTEST.COM'} at 127.0.0.1:37096)
I20250903 08:08:42.714931 19895 tablet_copy_service.cc:161] P ff08db2a004746a6ab7e1d0fa91eed38: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 094ea437f04c45fb93ad96244e3ca8a7 at {username='oryx', principal='oryx/127.12.201.253@KRBTEST.COM'} at 127.0.0.1:37096: session id = 094ea437f04c45fb93ad96244e3ca8a7-00000000000000000000000000000000
I20250903 08:08:42.716573 19895 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P ff08db2a004746a6ab7e1d0fa91eed38: Tablet Copy: opened 0 blocks and 1 log segments
I20250903 08:08:42.718309 20347 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.201.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.12.201.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/krb5kdc/oryx.keytab.
I20250903 08:08:42.719343 13095 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUML90r/build/debug/bin/kudu
/tmp/dist-test-taskUML90r/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.201.252:35949
--webserver_interface=127.12.201.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/krb5kdc/oryx.keytab
--principal=oryx/127.12.201.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.201.212:37903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.201.254:34143,127.12.201.253:42205,127.12.201.252:35949
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20250903 08:08:42.720731 20347 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P 094ea437f04c45fb93ad96244e3ca8a7: tablet copy: Starting download of 0 data blocks...
I20250903 08:08:42.720908 20347 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P 094ea437f04c45fb93ad96244e3ca8a7: tablet copy: Starting download of 1 WAL segments...
I20250903 08:08:42.723130 20347 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P 094ea437f04c45fb93ad96244e3ca8a7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250903 08:08:42.725164 19895 tablet_copy_service.cc:342] P ff08db2a004746a6ab7e1d0fa91eed38: Request end of tablet copy session 094ea437f04c45fb93ad96244e3ca8a7-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.12.201.253@KRBTEST.COM'} at 127.0.0.1:37096
I20250903 08:08:42.725328 19895 tablet_copy_service.cc:434] P ff08db2a004746a6ab7e1d0fa91eed38: ending tablet copy session 094ea437f04c45fb93ad96244e3ca8a7-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 094ea437f04c45fb93ad96244e3ca8a7
W20250903 08:08:42.731274 20347 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20250903 08:08:42.733741 20347 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:08:42.735441 20463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:42.735659 20460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:08:42.735898 20347 server_base.cc:1047] running on GCE node
W20250903 08:08:42.736085 20461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:08:42.736318 20347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:08:42.736596 20347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:08:42.737722 20347 hybrid_clock.cc:648] HybridClock initialized: now 1756886922737709 us; error 37 us; skew 500 ppm
Sep 03 08:08:42 dist-test-slave-27r9 krb5kdc[19827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886922, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.201.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 08:08:42.739472 20347 init.cc:377] Logged in from keytab as oryx/127.12.201.253@KRBTEST.COM (short username oryx)
I20250903 08:08:42.739980 20347 webserver.cc:480] Webserver started at http://127.12.201.253:36635/ using document root <none> and password file <none>
I20250903 08:08:42.740171 20347 fs_manager.cc:362] Metadata directory not provided
I20250903 08:08:42.740222 20347 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:08:42.741459 20347 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20250903 08:08:42.742410 20470 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:42.742703 20347 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250903 08:08:42.742777 20347 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/master-1/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/master-1/wal
uuid: "094ea437f04c45fb93ad96244e3ca8a7"
format_stamp: "Formatted at 2025-09-03 08:08:42 on dist-test-slave-27r9"
server_key: "da3ee0d928ee66e2b2bfcabdc086cad2"
server_key_iv: "2a7ee5998c0130360df073d3e56d2429"
server_key_version: "encryptionkey@0"
I20250903 08:08:42.742854 20347 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:08:42.755503 20347 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:08:42.757508 20347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:08:42.761955 20347 rpc_server.cc:307] RPC server started. Bound to: 127.12.201.253:42205
I20250903 08:08:42.762128 20522 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.201.253:42205 every 8 connection(s)
I20250903 08:08:42.762548 20347 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/master-1/data/info.pb
I20250903 08:08:42.764249 20523 sys_catalog.cc:263] Verifying existing consensus state
I20250903 08:08:42.765118 20523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 094ea437f04c45fb93ad96244e3ca8a7: Bootstrap starting.
I20250903 08:08:42.774837 20523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 094ea437f04c45fb93ad96244e3ca8a7: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250903 08:08:42.775159 20523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 094ea437f04c45fb93ad96244e3ca8a7: Bootstrap complete.
I20250903 08:08:42.775810 20523 raft_consensus.cc:357] T 00000000000000000000000000000000 P 094ea437f04c45fb93ad96244e3ca8a7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ff08db2a004746a6ab7e1d0fa91eed38" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 34143 } } peers { permanent_uuid: "094ea437f04c45fb93ad96244e3ca8a7" member_type: NON_VOTER last_known_addr { host: "127.12.201.253" port: 42205 } attrs { promote: true } }
I20250903 08:08:42.775945 20523 raft_consensus.cc:738] T 00000000000000000000000000000000 P 094ea437f04c45fb93ad96244e3ca8a7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 094ea437f04c45fb93ad96244e3ca8a7, State: Initialized, Role: LEARNER
I20250903 08:08:42.776060 20523 consensus_queue.cc:260] T 00000000000000000000000000000000 P 094ea437f04c45fb93ad96244e3ca8a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ff08db2a004746a6ab7e1d0fa91eed38" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 34143 } } peers { permanent_uuid: "094ea437f04c45fb93ad96244e3ca8a7" member_type: NON_VOTER last_known_addr { host: "127.12.201.253" port: 42205 } attrs { promote: true } }
I20250903 08:08:42.776394 20526 sys_catalog.cc:455] T 00000000000000000000000000000000 P 094ea437f04c45fb93ad96244e3ca8a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ff08db2a004746a6ab7e1d0fa91eed38" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 34143 } } peers { permanent_uuid: "094ea437f04c45fb93ad96244e3ca8a7" member_type: NON_VOTER last_known_addr { host: "127.12.201.253" port: 42205 } attrs { promote: true } } }
I20250903 08:08:42.776535 20523 sys_catalog.cc:564] T 00000000000000000000000000000000 P 094ea437f04c45fb93ad96244e3ca8a7 [sys.catalog]: configured and running, proceeding with master startup.
I20250903 08:08:42.776562 20526 sys_catalog.cc:458] T 00000000000000000000000000000000 P 094ea437f04c45fb93ad96244e3ca8a7 [sys.catalog]: This master's current role is: LEARNER
I20250903 08:08:42.780526 20537 catalog_manager.cc:1261] Loaded cluster ID: b7b764f24fd443ee92ab54013d2600ab
I20250903 08:08:42.780586 20537 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 094ea437f04c45fb93ad96244e3ca8a7: loading cluster ID for follower catalog manager: success
I20250903 08:08:42.782650 20537 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 094ea437f04c45fb93ad96244e3ca8a7: acquiring CA information for follower catalog manager: success
I20250903 08:08:42.784099 20537 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 094ea437f04c45fb93ad96244e3ca8a7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20250903 08:08:42.838390 20459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:08:42.838639 20459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:08:42.838693 20459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:08:42.841974 20459 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250903 08:08:42.842046 20459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 08:08:42.842069 20459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:08:42.842089 20459 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250903 08:08:42.842116 20459 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250903 08:08:42.846493 20459 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.201.212:37903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.12.201.254:34143,127.12.201.253:42205,127.12.201.252:35949
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/krb5kdc/oryx.keytab
--principal=oryx/127.12.201.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.201.252:35949
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.12.201.252
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20459
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 07:50:08 UTC on 5fd53c4cbb9d
build id 7874
I20250903 08:08:42.847720 20459 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:08:42.848534 20459 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:08:42.853945 20543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:42.853911 20545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:42.853935 20542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:08:42.854377 20459 server_base.cc:1047] running on GCE node
I20250903 08:08:42.854878 20459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:08:42.855808 20459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:08:42.856964 20459 hybrid_clock.cc:648] HybridClock initialized: now 1756886922856953 us; error 31 us; skew 500 ppm
Sep 03 08:08:42 dist-test-slave-27r9 krb5kdc[19827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886922, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.201.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 08:08:42.859609 20459 init.cc:377] Logged in from keytab as oryx/127.12.201.252@KRBTEST.COM (short username oryx)
I20250903 08:08:42.860620 20459 webserver.cc:480] Webserver started at http://127.12.201.252:34421/ using document root <none> and password file <none>
I20250903 08:08:42.861151 20459 fs_manager.cc:362] Metadata directory not provided
I20250903 08:08:42.861204 20459 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:08:42.861413 20459 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:08:42.863127 20459 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/master-2/data/instance:
uuid: "0fba560828064814bed66061ed3ebc3d"
format_stamp: "Formatted at 2025-09-03 08:08:42 on dist-test-slave-27r9"
server_key: "457b70f2ae351abb894ac9894739b423"
server_key_iv: "ae39ed13e87a046341c973b3a183afa2"
server_key_version: "encryptionkey@0"
I20250903 08:08:42.863668 20459 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/master-2/wal/instance:
uuid: "0fba560828064814bed66061ed3ebc3d"
format_stamp: "Formatted at 2025-09-03 08:08:42 on dist-test-slave-27r9"
server_key: "457b70f2ae351abb894ac9894739b423"
server_key_iv: "ae39ed13e87a046341c973b3a183afa2"
server_key_version: "encryptionkey@0"
I20250903 08:08:42.866953 20459 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20250903 08:08:42.869127 20552 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:42.870151 20459 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20250903 08:08:42.870285 20459 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/master-2/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/master-2/wal
uuid: "0fba560828064814bed66061ed3ebc3d"
format_stamp: "Formatted at 2025-09-03 08:08:42 on dist-test-slave-27r9"
server_key: "457b70f2ae351abb894ac9894739b423"
server_key_iv: "ae39ed13e87a046341c973b3a183afa2"
server_key_version: "encryptionkey@0"
I20250903 08:08:42.870378 20459 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:08:42.881960 20459 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:08:42.884486 20459 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:08:42.884658 20459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:08:42.891377 20604 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.201.252:35949 every 8 connection(s)
I20250903 08:08:42.891376 20459 rpc_server.cc:307] RPC server started. Bound to: 127.12.201.252:35949
I20250903 08:08:42.892405 20459 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/master-2/data/info.pb
I20250903 08:08:42.894837 20605 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:08:42.895502 13095 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUML90r/build/debug/bin/kudu as pid 20459
I20250903 08:08:42.896389 20068 heartbeater.cc:499] Master 127.12.201.254:34143 was elected leader, sending a full tablet report...
I20250903 08:08:42.899379 20605 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.201.254" port: 34143 } has no permanent_uuid. Determining permanent_uuid...
Sep 03 08:08:42 dist-test-slave-27r9 krb5kdc[19827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.201.252@KRBTEST.COM
Sep 03 08:08:42 dist-test-slave-27r9 krb5kdc[19827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886922, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.201.252@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM
I20250903 08:08:42.909478 20608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:42.900742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:34143 (local address 127.0.0.1:37108)
0903 08:08:42.901055 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:42.901096 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:42.901215 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:42.901525 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:42.901531 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:42.901959 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:08:42.902163 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:42.902169 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:42.903262 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:42.903265 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:42.903848 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:42.903857 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:42.903970 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:42.904476 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:08:42.904496 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:08:42.905943 (+ 1447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:08:42.907976 (+ 2033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:42.907983 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:42.907987 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:42.908247 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:42.908485 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:42.908490 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:42.908493 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:42.908535 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:42.908995 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:08:42.909003 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:08:42.909090 (+ 87us) client_negotiation.cc:769] Sending connection context
0903 08:08:42.909196 (+ 106us) client_negotiation.cc:240] Negotiation successful
0903 08:08:42.909313 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"spinlock_wait_cycles":2944,"thread_start_us":92,"threads_started":1}
I20250903 08:08:42.909500 20607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:42.897236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56842 (local address 127.12.201.252:35949)
0903 08:08:42.897748 (+ 512us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:42.897757 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:42.897790 (+ 33us) server_negotiation.cc:408] Connection header received
0903 08:08:42.898311 (+ 521us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:42.898326 (+ 15us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:42.898608 (+ 282us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:42.898901 (+ 293us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:42.899879 (+ 978us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:42.900629 (+ 750us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:42.901401 (+ 772us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:42.901686 (+ 285us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:42.904319 (+ 2633us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:42.904341 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:42.904356 (+ 15us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:08:42.904406 (+ 50us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:42.906684 (+ 2278us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:42.907165 (+ 481us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:42.907170 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:42.907175 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:42.907264 (+ 89us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:42.907534 (+ 270us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:42.907538 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:42.907540 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:42.908024 (+ 484us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:42.908233 (+ 209us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:42.908529 (+ 296us) server_negotiation.cc:300] Negotiation successful
0903 08:08:42.908792 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":145,"threads_started":1}
I20250903 08:08:42.909530 19914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:42.900905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37108 (local address 127.12.201.254:34143)
0903 08:08:42.901032 (+ 127us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:42.901036 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:42.901172 (+ 136us) server_negotiation.cc:408] Connection header received
0903 08:08:42.901320 (+ 148us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:42.901323 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:42.901382 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:42.901457 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:42.902292 (+ 835us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:42.903159 (+ 867us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:42.903956 (+ 797us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:42.904158 (+ 202us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:42.906078 (+ 1920us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:42.906098 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:42.906101 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:08:42.906132 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:42.907840 (+ 1708us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:42.908356 (+ 516us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:42.908359 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:42.908361 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:42.908402 (+ 41us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:42.908695 (+ 293us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:42.908699 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:42.908701 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:42.908859 (+ 158us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:42.908918 (+ 59us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:42.909279 (+ 361us) server_negotiation.cc:300] Negotiation successful
0903 08:08:42.909385 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20250903 08:08:42.910833 20605 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.201.253" port: 42205 } has no permanent_uuid. Determining permanent_uuid...
Sep 03 08:08:42 dist-test-slave-27r9 krb5kdc[19827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886922, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.201.252@KRBTEST.COM for oryx/127.12.201.253@KRBTEST.COM
I20250903 08:08:42.917894 20608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:42.911308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.253:42205 (local address 127.0.0.1:38724)
0903 08:08:42.911470 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:42.911486 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:42.911543 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:42.911920 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:42.911922 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:42.912063 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:08:42.912244 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:42.912250 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:42.913117 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:42.913120 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:42.913511 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:42.913518 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:42.913607 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:42.914072 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:08:42.914084 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:08:42.915225 (+ 1141us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:08:42.916892 (+ 1667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:42.916895 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:42.916896 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:42.917053 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:42.917250 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:42.917253 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:42.917254 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:42.917296 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:42.917549 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:08:42.917550 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:08:42.917613 (+ 63us) client_negotiation.cc:769] Sending connection context
0903 08:08:42.917684 (+ 71us) client_negotiation.cc:240] Negotiation successful
0903 08:08:42.917772 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95}
I20250903 08:08:42.917950 20609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:42.911442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38724 (local address 127.12.201.253:42205)
0903 08:08:42.911691 (+ 249us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:42.911694 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:42.911706 (+ 12us) server_negotiation.cc:408] Connection header received
0903 08:08:42.911742 (+ 36us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:42.911744 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:42.911784 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:42.911863 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:42.912348 (+ 485us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:42.913011 (+ 663us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:42.913600 (+ 589us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:42.913728 (+ 128us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:42.915349 (+ 1621us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:42.915367 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:42.915370 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:08:42.915395 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:42.916780 (+ 1385us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:42.917137 (+ 357us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:42.917140 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:42.917141 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:42.917178 (+ 37us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:42.917360 (+ 182us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:42.917362 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:42.917364 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:42.917488 (+ 124us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:42.917562 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:42.917724 (+ 162us) server_negotiation.cc:300] Negotiation successful
0903 08:08:42.917818 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":93,"threads_started":1}
I20250903 08:08:42.918695 20605 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.201.252" port: 35949 } has no permanent_uuid. Determining permanent_uuid...
Sep 03 08:08:42 dist-test-slave-27r9 krb5kdc[19827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886922, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.201.252@KRBTEST.COM for oryx/127.12.201.252@KRBTEST.COM
I20250903 08:08:42.925196 20608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:42.919127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.252:35949 (local address 127.0.0.1:56844)
0903 08:08:42.919243 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:42.919256 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:42.919331 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:42.919547 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:42.919550 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:42.919686 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:08:42.919818 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:42.919823 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:42.920568 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:42.920571 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:42.920880 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:42.920886 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:42.920982 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:42.921429 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:08:42.921441 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:08:42.922466 (+ 1025us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:08:42.924074 (+ 1608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:42.924078 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:42.924080 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:42.924229 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:42.924431 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:42.924434 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:42.924438 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:42.924474 (+ 36us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:42.924753 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:08:42.924756 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:08:42.924822 (+ 66us) client_negotiation.cc:769] Sending connection context
0903 08:08:42.924905 (+ 83us) client_negotiation.cc:240] Negotiation successful
0903 08:08:42.925026 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20250903 08:08:42.925207 20607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:42.919238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56844 (local address 127.12.201.252:35949)
0903 08:08:42.919332 (+ 94us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:42.919336 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:42.919349 (+ 13us) server_negotiation.cc:408] Connection header received
0903 08:08:42.919417 (+ 68us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:42.919421 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:42.919462 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:42.919529 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:42.919939 (+ 410us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:42.920471 (+ 532us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:42.920975 (+ 504us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:42.921183 (+ 208us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:42.922576 (+ 1393us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:42.922588 (+ 12us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:42.922590 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:08:42.922609 (+ 19us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:42.923978 (+ 1369us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:42.924316 (+ 338us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:42.924319 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:42.924324 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:42.924356 (+ 32us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:42.924551 (+ 195us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:42.924554 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:42.924556 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:42.924673 (+ 117us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:42.924755 (+ 82us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:42.924951 (+ 196us) server_negotiation.cc:300] Negotiation successful
0903 08:08:42.925079 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":37}
I20250903 08:08:42.927949 20605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fba560828064814bed66061ed3ebc3d: Bootstrap starting.
I20250903 08:08:42.930369 20605 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0fba560828064814bed66061ed3ebc3d: Neither blocks nor log segments found. Creating new log.
I20250903 08:08:42.931151 20605 log.cc:826] T 00000000000000000000000000000000 P 0fba560828064814bed66061ed3ebc3d: Log is configured to *not* fsync() on all Append() calls
I20250903 08:08:42.932835 20605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fba560828064814bed66061ed3ebc3d: No bootstrap required, opened a new log
I20250903 08:08:42.935196 20605 raft_consensus.cc:357] T 00000000000000000000000000000000 P 0fba560828064814bed66061ed3ebc3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff08db2a004746a6ab7e1d0fa91eed38" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 34143 } } peers { permanent_uuid: "094ea437f04c45fb93ad96244e3ca8a7" member_type: VOTER last_known_addr { host: "127.12.201.253" port: 42205 } } peers { permanent_uuid: "0fba560828064814bed66061ed3ebc3d" member_type: VOTER last_known_addr { host: "127.12.201.252" port: 35949 } }
I20250903 08:08:42.935374 20605 raft_consensus.cc:383] T 00000000000000000000000000000000 P 0fba560828064814bed66061ed3ebc3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:08:42.935434 20605 raft_consensus.cc:738] T 00000000000000000000000000000000 P 0fba560828064814bed66061ed3ebc3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fba560828064814bed66061ed3ebc3d, State: Initialized, Role: FOLLOWER
I20250903 08:08:42.935832 20605 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0fba560828064814bed66061ed3ebc3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff08db2a004746a6ab7e1d0fa91eed38" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 34143 } } peers { permanent_uuid: "094ea437f04c45fb93ad96244e3ca8a7" member_type: VOTER last_known_addr { host: "127.12.201.253" port: 42205 } } peers { permanent_uuid: "0fba560828064814bed66061ed3ebc3d" member_type: VOTER last_known_addr { host: "127.12.201.252" port: 35949 } }
I20250903 08:08:42.936259 20611 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fba560828064814bed66061ed3ebc3d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff08db2a004746a6ab7e1d0fa91eed38" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 34143 } } peers { permanent_uuid: "094ea437f04c45fb93ad96244e3ca8a7" member_type: VOTER last_known_addr { host: "127.12.201.253" port: 42205 } } peers { permanent_uuid: "0fba560828064814bed66061ed3ebc3d" member_type: VOTER last_known_addr { host: "127.12.201.252" port: 35949 } } }
I20250903 08:08:42.936365 20611 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fba560828064814bed66061ed3ebc3d [sys.catalog]: This master's current role is: FOLLOWER
I20250903 08:08:42.936892 20605 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0fba560828064814bed66061ed3ebc3d [sys.catalog]: configured and running, proceeding with master startup.
W20250903 08:08:42.941267 20622 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 0fba560828064814bed66061ed3ebc3d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250903 08:08:42.941380 20622 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250903 08:08:42.944583 20459 master_runner.cc:428] Detected that this master 0fba560828064814bed66061ed3ebc3d is joining an existing cluster
I20250903 08:08:42.944635 20459 master_runner.cc:432] Initiating AddMaster RPC to add 127.12.201.252:35949
I20250903 08:08:42.966776 20440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:42.963821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:34143 (local address 127.0.0.1:37130)
0903 08:08:42.963951 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:42.963966 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:42.964065 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:42.964567 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:42.964569 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:42.964588 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 08:08:42.964763 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:42.964767 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:42.965821 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:42.965824 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:42.966398 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:42.966404 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:42.966492 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:42.966538 (+ 46us) client_negotiation.cc:769] Sending connection context
0903 08:08:42.966586 (+ 48us) client_negotiation.cc:240] Negotiation successful
0903 08:08:42.966636 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20250903 08:08:42.967705 20631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:42.963969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37130 (local address 127.12.201.254:34143)
0903 08:08:42.964331 (+ 362us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:42.964335 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:42.964350 (+ 15us) server_negotiation.cc:408] Connection header received
0903 08:08:42.964404 (+ 54us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:42.964408 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:42.964454 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:42.964553 (+ 99us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 08:08:42.964886 (+ 333us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:42.965730 (+ 844us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:42.966509 (+ 779us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:42.967030 (+ 521us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:42.967441 (+ 411us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:42.967512 (+ 71us) server_negotiation.cc:300] Negotiation successful
0903 08:08:42.967584 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":146,"threads_started":1}
I20250903 08:08:42.975343 20630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:42.947199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.252:35949 (local address 127.0.0.1:56850)
0903 08:08:42.951187 (+ 3988us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:42.951201 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:42.951306 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:42.951523 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:42.951527 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:42.967367 (+ 15840us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:08:42.967758 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:42.967772 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:42.968695 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:42.968698 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:42.969259 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:42.969268 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:42.969374 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:42.969978 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:08:42.969993 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:08:42.970479 (+ 486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:08:42.972872 (+ 2393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:42.972875 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:42.972878 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:42.973624 (+ 746us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:42.973895 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:42.973900 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:42.973904 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:42.973965 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:42.974728 (+ 763us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:08:42.974734 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:08:42.974825 (+ 91us) client_negotiation.cc:769] Sending connection context
0903 08:08:42.974966 (+ 141us) client_negotiation.cc:240] Negotiation successful
0903 08:08:42.975113 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3905,"spinlock_wait_cycles":5248,"thread_start_us":81,"threads_started":1}
I20250903 08:08:42.975805 20607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:42.947009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56850 (local address 127.12.201.252:35949)
0903 08:08:42.947132 (+ 123us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:42.947138 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:42.951276 (+ 4138us) server_negotiation.cc:408] Connection header received
0903 08:08:42.951384 (+ 108us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:42.951386 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:42.951423 (+ 37us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:42.951486 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:42.967925 (+ 16439us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:42.968576 (+ 651us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:42.969624 (+ 1048us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:42.969829 (+ 205us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:42.970615 (+ 786us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:42.970630 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:42.970634 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:08:42.970656 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:42.972759 (+ 2103us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:42.973733 (+ 974us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:42.973736 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:42.973738 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:42.973792 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:42.974059 (+ 267us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:42.974064 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:42.974067 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:42.974524 (+ 457us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:42.974617 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:42.975538 (+ 921us) server_negotiation.cc:300] Negotiation successful
0903 08:08:42.975656 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20250903 08:08:42.976867 20440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:42.970603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.253:42205 (local address 127.0.0.1:38740)
0903 08:08:42.971829 (+ 1226us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:42.971843 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:42.971928 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:42.973126 (+ 1198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:42.973132 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:42.973158 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 08:08:42.973390 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:42.973399 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:42.975625 (+ 2226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:42.975629 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:42.976541 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:42.976550 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:42.976633 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:42.976650 (+ 17us) client_negotiation.cc:769] Sending connection context
0903 08:08:42.976688 (+ 38us) client_negotiation.cc:240] Negotiation successful
0903 08:08:42.976744 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1145}
I20250903 08:08:42.977025 20628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:42.946596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.253:42205 (local address 127.0.0.1:38728)
0903 08:08:42.947127 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:42.947140 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:42.947265 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:42.947626 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:42.947629 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:42.967356 (+ 19727us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:08:42.968063 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:42.968070 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:42.969149 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:42.969153 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:42.969754 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:42.969763 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:42.969866 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:42.971387 (+ 1521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:08:42.971402 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:08:42.972081 (+ 679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:08:42.974192 (+ 2111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:42.974196 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:42.974198 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:42.974408 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:42.976033 (+ 1625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:42.976039 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:42.976044 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:42.976101 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:42.976533 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:08:42.976536 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:08:42.976617 (+ 81us) client_negotiation.cc:769] Sending connection context
0903 08:08:42.976704 (+ 87us) client_negotiation.cc:240] Negotiation successful
0903 08:08:42.976819 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"spinlock_wait_cycles":53632,"thread_start_us":96,"threads_started":1}
I20250903 08:08:42.977504 20609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:42.946802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38728 (local address 127.12.201.253:42205)
0903 08:08:42.946929 (+ 127us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:42.946934 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:42.947204 (+ 270us) server_negotiation.cc:408] Connection header received
0903 08:08:42.947372 (+ 168us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:42.947378 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:42.947426 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:42.947476 (+ 50us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:42.968216 (+ 20740us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:42.969004 (+ 788us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:42.971032 (+ 2028us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:42.971239 (+ 207us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:42.972202 (+ 963us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:42.972222 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:42.972225 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:08:42.972251 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:42.974071 (+ 1820us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:42.974531 (+ 460us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:42.974538 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:42.974541 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:42.974607 (+ 66us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:42.976207 (+ 1600us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:42.976212 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:42.976214 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:42.976401 (+ 187us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:42.976488 (+ 87us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:42.977233 (+ 745us) server_negotiation.cc:300] Negotiation successful
0903 08:08:42.977353 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20250903 08:08:42.977901 19914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:42.946564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37118 (local address 127.12.201.254:34143)
0903 08:08:42.946691 (+ 127us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:42.946696 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:42.946963 (+ 267us) server_negotiation.cc:408] Connection header received
0903 08:08:42.947178 (+ 215us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:42.947182 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:42.947237 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:42.947308 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:42.968705 (+ 21397us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:42.969411 (+ 706us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:42.971183 (+ 1772us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:42.971698 (+ 515us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:42.973525 (+ 1827us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:42.973543 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:42.973548 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:08:42.973707 (+ 159us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:42.975553 (+ 1846us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:42.976408 (+ 855us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:42.976412 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:42.976414 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:42.976463 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:42.976717 (+ 254us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:42.976720 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:42.976722 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:42.976878 (+ 156us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:42.976959 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:42.977623 (+ 664us) server_negotiation.cc:300] Negotiation successful
0903 08:08:42.977735 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20250903 08:08:42.978001 20632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:42.970827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38740 (local address 127.12.201.253:42205)
0903 08:08:42.971570 (+ 743us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:42.971578 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:42.972553 (+ 975us) server_negotiation.cc:408] Connection header received
0903 08:08:42.972622 (+ 69us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:42.972626 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:42.972684 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:42.973098 (+ 414us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 08:08:42.974716 (+ 1618us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:42.975485 (+ 769us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:42.977151 (+ 1666us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:42.977639 (+ 488us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:42.977780 (+ 141us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:42.977834 (+ 54us) server_negotiation.cc:300] Negotiation successful
0903 08:08:42.977888 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":99,"threads_started":1}
I20250903 08:08:42.979111 20629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:42.946464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:34143 (local address 127.0.0.1:37118)
0903 08:08:42.946885 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:42.946901 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:42.947059 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:42.947450 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:42.947453 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:42.967373 (+ 19920us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:08:42.968576 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:42.968582 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:42.969552 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:42.969555 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:42.970732 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:42.970740 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:42.971829 (+ 1089us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:42.972343 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:08:42.972363 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:08:42.973395 (+ 1032us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:08:42.976089 (+ 2694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:42.976092 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:42.976094 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:42.976291 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:42.976559 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:42.976562 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:42.976564 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:42.976613 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:42.977350 (+ 737us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:08:42.977353 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:08:42.977426 (+ 73us) client_negotiation.cc:769] Sending connection context
0903 08:08:42.978777 (+ 1351us) client_negotiation.cc:240] Negotiation successful
0903 08:08:42.978901 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"spinlock_wait_cycles":45312,"thread_start_us":114,"threads_started":1}
Sep 03 08:08:42 dist-test-slave-27r9 krb5kdc[19827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886921, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.201.254@KRBTEST.COM for oryx/127.12.201.252@KRBTEST.COM
I20250903 08:08:42.990084 20440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:42.982607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.252:35949 (local address 127.0.0.1:56862)
0903 08:08:42.982703 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:42.982717 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:42.982792 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:42.983166 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:42.983169 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:42.983350 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:08:42.983575 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:42.983582 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:42.984468 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:42.984471 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:42.984801 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:42.984808 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:42.984896 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:42.985330 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:08:42.985343 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:08:42.986778 (+ 1435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:08:42.988796 (+ 2018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:42.988799 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:42.988802 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:42.988967 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:42.989178 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:42.989183 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:42.989185 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:42.989227 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:42.989630 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:08:42.989633 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:08:42.989715 (+ 82us) client_negotiation.cc:769] Sending connection context
0903 08:08:42.989805 (+ 90us) client_negotiation.cc:240] Negotiation successful
0903 08:08:42.989929 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20250903 08:08:42.990087 20607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:42.982676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56862 (local address 127.12.201.252:35949)
0903 08:08:42.982788 (+ 112us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:42.982793 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:42.982806 (+ 13us) server_negotiation.cc:408] Connection header received
0903 08:08:42.982917 (+ 111us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:42.982920 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:42.982971 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:42.983063 (+ 92us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:42.983704 (+ 641us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:42.984367 (+ 663us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:42.984904 (+ 537us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:42.985085 (+ 181us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:42.986914 (+ 1829us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:42.986927 (+ 13us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:42.986930 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:08:42.986950 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:42.988694 (+ 1744us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:42.989057 (+ 363us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:42.989061 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:42.989062 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:42.989099 (+ 37us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:42.989311 (+ 212us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:42.989315 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:42.989317 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:42.989491 (+ 174us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:42.989585 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:42.989866 (+ 281us) server_negotiation.cc:300] Negotiation successful
0903 08:08:42.989962 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20250903 08:08:42.991284 19873 catalog_manager.cc:7069] Initiating ChangeConfig request to add master 127.12.201.252:35949: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "0fba560828064814bed66061ed3ebc3d"
member_type: NON_VOTER
last_known_addr {
host: "127.12.201.252"
port: 35949
}
attrs {
promote: true
}
}
dest_uuid: "ff08db2a004746a6ab7e1d0fa91eed38"
cas_config_opid_index: 5
I20250903 08:08:42.991808 19873 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff08db2a004746a6ab7e1d0fa91eed38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ff08db2a004746a6ab7e1d0fa91eed38" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 34143 } } peers { permanent_uuid: "094ea437f04c45fb93ad96244e3ca8a7" member_type: NON_VOTER last_known_addr { host: "127.12.201.253" port: 42205 } attrs { promote: true } } peers { permanent_uuid: "0fba560828064814bed66061ed3ebc3d" member_type: NON_VOTER last_known_addr { host: "127.12.201.252" port: 35949 } attrs { promote: true } }
I20250903 08:08:42.993076 20445 raft_consensus.cc:2953] T 00000000000000000000000000000000 P ff08db2a004746a6ab7e1d0fa91eed38 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 0fba560828064814bed66061ed3ebc3d (127.12.201.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ff08db2a004746a6ab7e1d0fa91eed38" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 34143 } } peers { permanent_uuid: "094ea437f04c45fb93ad96244e3ca8a7" member_type: NON_VOTER last_known_addr { host: "127.12.201.253" port: 42205 } attrs { promote: true } } peers { permanent_uuid: "0fba560828064814bed66061ed3ebc3d" member_type: NON_VOTER last_known_addr { host: "127.12.201.252" port: 35949 } attrs { promote: true } } }
I20250903 08:08:42.994257 20445 catalog_manager.cc:7073] Successfully completed master ChangeConfig request to add master 127.12.201.252:35949
I20250903 08:08:42.994405 20446 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff08db2a004746a6ab7e1d0fa91eed38 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ff08db2a004746a6ab7e1d0fa91eed38" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ff08db2a004746a6ab7e1d0fa91eed38" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 34143 } } peers { permanent_uuid: "094ea437f04c45fb93ad96244e3ca8a7" member_type: NON_VOTER last_known_addr { host: "127.12.201.253" port: 42205 } attrs { promote: true } } peers { permanent_uuid: "0fba560828064814bed66061ed3ebc3d" member_type: NON_VOTER last_known_addr { host: "127.12.201.252" port: 35949 } attrs { promote: true } } }
I20250903 08:08:42.994503 20446 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff08db2a004746a6ab7e1d0fa91eed38 [sys.catalog]: This master's current role is: LEADER
I20250903 08:08:42.994947 20497 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 094ea437f04c45fb93ad96244e3ca8a7 [term 1 LEARNER]: Refusing update from remote peer ff08db2a004746a6ab7e1d0fa91eed38: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250903 08:08:42.995136 20579 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 0fba560828064814bed66061ed3ebc3d [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:08:42.995229 20459 master.cc:561] Master@127.12.201.252:35949 shutting down...
I20250903 08:08:42.996057 20446 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ff08db2a004746a6ab7e1d0fa91eed38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "094ea437f04c45fb93ad96244e3ca8a7" member_type: NON_VOTER last_known_addr { host: "127.12.201.253" port: 42205 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250903 08:08:42.996534 20579 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 0fba560828064814bed66061ed3ebc3d [term 1 FOLLOWER]: Refusing update from remote peer ff08db2a004746a6ab7e1d0fa91eed38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250903 08:08:42.997202 20446 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ff08db2a004746a6ab7e1d0fa91eed38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fba560828064814bed66061ed3ebc3d" member_type: NON_VOTER last_known_addr { host: "127.12.201.252" port: 35949 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 08:08:42.998363 20497 raft_consensus.cc:2953] T 00000000000000000000000000000000 P 094ea437f04c45fb93ad96244e3ca8a7 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 0fba560828064814bed66061ed3ebc3d (127.12.201.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ff08db2a004746a6ab7e1d0fa91eed38" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 34143 } } peers { permanent_uuid: "094ea437f04c45fb93ad96244e3ca8a7" member_type: NON_VOTER last_known_addr { host: "127.12.201.253" port: 42205 } attrs { promote: true } } peers { permanent_uuid: "0fba560828064814bed66061ed3ebc3d" member_type: NON_VOTER last_known_addr { host: "127.12.201.252" port: 35949 } attrs { promote: true } } }
W20250903 08:08:42.998903 19862 consensus_peers.cc:489] T 00000000000000000000000000000000 P ff08db2a004746a6ab7e1d0fa91eed38 -> Peer 0fba560828064814bed66061ed3ebc3d (127.12.201.252:35949): Couldn't send request to peer 0fba560828064814bed66061ed3ebc3d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20250903 08:08:42.999966 20634 sys_catalog.cc:455] T 00000000000000000000000000000000 P 094ea437f04c45fb93ad96244e3ca8a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff08db2a004746a6ab7e1d0fa91eed38. Latest consensus state: current_term: 1 leader_uuid: "ff08db2a004746a6ab7e1d0fa91eed38" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ff08db2a004746a6ab7e1d0fa91eed38" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 34143 } } peers { permanent_uuid: "094ea437f04c45fb93ad96244e3ca8a7" member_type: NON_VOTER last_known_addr { host: "127.12.201.253" port: 42205 } attrs { promote: true } } peers { permanent_uuid: "0fba560828064814bed66061ed3ebc3d" member_type: NON_VOTER last_known_addr { host: "127.12.201.252" port: 35949 } attrs { promote: true } } }
I20250903 08:08:43.000165 20634 sys_catalog.cc:458] T 00000000000000000000000000000000 P 094ea437f04c45fb93ad96244e3ca8a7 [sys.catalog]: This master's current role is: LEARNER
I20250903 08:08:43.000494 20445 raft_consensus.cc:1062] T 00000000000000000000000000000000 P ff08db2a004746a6ab7e1d0fa91eed38: attempting to promote NON_VOTER 094ea437f04c45fb93ad96244e3ca8a7 to VOTER
I20250903 08:08:43.000041 20526 sys_catalog.cc:455] T 00000000000000000000000000000000 P 094ea437f04c45fb93ad96244e3ca8a7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ff08db2a004746a6ab7e1d0fa91eed38" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ff08db2a004746a6ab7e1d0fa91eed38" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 34143 } } peers { permanent_uuid: "094ea437f04c45fb93ad96244e3ca8a7" member_type: NON_VOTER last_known_addr { host: "127.12.201.253" port: 42205 } attrs { promote: true } } peers { permanent_uuid: "0fba560828064814bed66061ed3ebc3d" member_type: NON_VOTER last_known_addr { host: "127.12.201.252" port: 35949 } attrs { promote: true } } }
I20250903 08:08:43.000794 20526 sys_catalog.cc:458] T 00000000000000000000000000000000 P 094ea437f04c45fb93ad96244e3ca8a7 [sys.catalog]: This master's current role is: LEARNER
I20250903 08:08:43.001264 20445 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff08db2a004746a6ab7e1d0fa91eed38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ff08db2a004746a6ab7e1d0fa91eed38" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 34143 } } peers { permanent_uuid: "094ea437f04c45fb93ad96244e3ca8a7" member_type: VOTER last_known_addr { host: "127.12.201.253" port: 42205 } attrs { promote: false } } peers { permanent_uuid: "0fba560828064814bed66061ed3ebc3d" member_type: NON_VOTER last_known_addr { host: "127.12.201.252" port: 35949 } attrs { promote: true } }
I20250903 08:08:43.002920 20459 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 0fba560828064814bed66061ed3ebc3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 08:08:43.003091 20497 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 094ea437f04c45fb93ad96244e3ca8a7 [term 1 LEARNER]: Refusing update from remote peer ff08db2a004746a6ab7e1d0fa91eed38: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250903 08:08:43.003341 20459 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 0fba560828064814bed66061ed3ebc3d [term 1 FOLLOWER]: Raft consensus is shut down!
W20250903 08:08:43.003470 19862 consensus_peers.cc:489] T 00000000000000000000000000000000 P ff08db2a004746a6ab7e1d0fa91eed38 -> Peer 0fba560828064814bed66061ed3ebc3d (127.12.201.252:35949): Couldn't send request to peer 0fba560828064814bed66061ed3ebc3d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20250903 08:08:43.003491 20459 tablet_replica.cc:331] T 00000000000000000000000000000000 P 0fba560828064814bed66061ed3ebc3d: stopping tablet replica
I20250903 08:08:43.003665 20446 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ff08db2a004746a6ab7e1d0fa91eed38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "094ea437f04c45fb93ad96244e3ca8a7" member_type: VOTER last_known_addr { host: "127.12.201.253" port: 42205 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250903 08:08:43.005240 20446 raft_consensus.cc:2953] T 00000000000000000000000000000000 P ff08db2a004746a6ab7e1d0fa91eed38 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 094ea437f04c45fb93ad96244e3ca8a7 (127.12.201.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ff08db2a004746a6ab7e1d0fa91eed38" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 34143 } } peers { permanent_uuid: "094ea437f04c45fb93ad96244e3ca8a7" member_type: VOTER last_known_addr { host: "127.12.201.253" port: 42205 } attrs { promote: false } } peers { permanent_uuid: "0fba560828064814bed66061ed3ebc3d" member_type: NON_VOTER last_known_addr { host: "127.12.201.252" port: 35949 } attrs { promote: true } } }
I20250903 08:08:43.006546 20446 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff08db2a004746a6ab7e1d0fa91eed38 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ff08db2a004746a6ab7e1d0fa91eed38" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ff08db2a004746a6ab7e1d0fa91eed38" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 34143 } } peers { permanent_uuid: "094ea437f04c45fb93ad96244e3ca8a7" member_type: VOTER last_known_addr { host: "127.12.201.253" port: 42205 } attrs { promote: false } } peers { permanent_uuid: "0fba560828064814bed66061ed3ebc3d" member_type: NON_VOTER last_known_addr { host: "127.12.201.252" port: 35949 } attrs { promote: true } } }
I20250903 08:08:43.006645 20446 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff08db2a004746a6ab7e1d0fa91eed38 [sys.catalog]: This master's current role is: LEADER
I20250903 08:08:43.006757 20446 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff08db2a004746a6ab7e1d0fa91eed38 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ff08db2a004746a6ab7e1d0fa91eed38" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ff08db2a004746a6ab7e1d0fa91eed38" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 34143 } } peers { permanent_uuid: "094ea437f04c45fb93ad96244e3ca8a7" member_type: VOTER last_known_addr { host: "127.12.201.253" port: 42205 } attrs { promote: false } } peers { permanent_uuid: "0fba560828064814bed66061ed3ebc3d" member_type: NON_VOTER last_known_addr { host: "127.12.201.252" port: 35949 } attrs { promote: true } } }
I20250903 08:08:43.006827 20446 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff08db2a004746a6ab7e1d0fa91eed38 [sys.catalog]: This master's current role is: LEADER
I20250903 08:08:43.007450 20497 raft_consensus.cc:2953] T 00000000000000000000000000000000 P 094ea437f04c45fb93ad96244e3ca8a7 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 094ea437f04c45fb93ad96244e3ca8a7 (127.12.201.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ff08db2a004746a6ab7e1d0fa91eed38" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 34143 } } peers { permanent_uuid: "094ea437f04c45fb93ad96244e3ca8a7" member_type: VOTER last_known_addr { host: "127.12.201.253" port: 42205 } attrs { promote: false } } peers { permanent_uuid: "0fba560828064814bed66061ed3ebc3d" member_type: NON_VOTER last_known_addr { host: "127.12.201.252" port: 35949 } attrs { promote: true } } }
I20250903 08:08:43.007964 20459 master.cc:583] Master@127.12.201.252:35949 shutdown complete.
I20250903 08:08:43.008033 20459 master_runner.cc:305] Clearing existing system tablet
I20250903 08:08:43.008323 20459 fs_manager.cc:362] Metadata directory not provided
I20250903 08:08:43.008394 20459 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:08:43.008757 20526 sys_catalog.cc:455] T 00000000000000000000000000000000 P 094ea437f04c45fb93ad96244e3ca8a7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ff08db2a004746a6ab7e1d0fa91eed38" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ff08db2a004746a6ab7e1d0fa91eed38" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 34143 } } peers { permanent_uuid: "094ea437f04c45fb93ad96244e3ca8a7" member_type: VOTER last_known_addr { host: "127.12.201.253" port: 42205 } attrs { promote: false } } peers { permanent_uuid: "0fba560828064814bed66061ed3ebc3d" member_type: NON_VOTER last_known_addr { host: "127.12.201.252" port: 35949 } attrs { promote: true } } }
I20250903 08:08:43.008857 20526 sys_catalog.cc:458] T 00000000000000000000000000000000 P 094ea437f04c45fb93ad96244e3ca8a7 [sys.catalog]: This master's current role is: FOLLOWER
I20250903 08:08:43.010049 20459 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20250903 08:08:43.010468 20440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:43.010173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.252:35949 (local address 127.0.0.1:56866)
0903 08:08:43.010303 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:43.010380 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.201.252:35949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20250903 08:08:43.010704 19873 master.cc:679] Network error: unable to get registration information for peer 0fba560828064814bed66061ed3ebc3d: Client connection negotiation failed: client connection to 127.12.201.252:35949: connect: Connection refused (error 111)
I20250903 08:08:43.011021 20636 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:43.011835 20459 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:08:43.011947 20459 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250903 08:08:43.011984 20459 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/master-2/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/master-2/wal
uuid: "0fba560828064814bed66061ed3ebc3d"
format_stamp: "Formatted at 2025-09-03 08:08:42 on dist-test-slave-27r9"
server_key: "457b70f2ae351abb894ac9894739b423"
server_key_iv: "ae39ed13e87a046341c973b3a183afa2"
server_key_version: "encryptionkey@0"
I20250903 08:08:43.013530 20459 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 0fba560828064814bed66061ed3ebc3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 08:08:43.015311 20459 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 0fba560828064814bed66061ed3ebc3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20250903 08:08:43.015379 20459 log.cc:1199] T 00000000000000000000000000000000 P 0fba560828064814bed66061ed3ebc3d: Deleting WAL directory at /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20250903 08:08:43.015588 20459 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 0fba560828064814bed66061ed3ebc3d: Deleting consensus metadata
I20250903 08:08:43.015813 20459 master_runner.cc:315] Copying system tablet from 127.12.201.254:34143
I20250903 08:08:43.016875 20459 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P 0fba560828064814bed66061ed3ebc3d: tablet copy: Beginning tablet copy session from remote peer at address 127.12.201.254:34143
I20250903 08:08:43.023636 20642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:43.017331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:34143 (local address 127.0.0.1:37140)
0903 08:08:43.017654 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:43.017667 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:43.017774 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:43.018009 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:43.018012 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:43.018205 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:08:43.018395 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:43.018401 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:43.019138 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:43.019142 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:43.019514 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:43.019522 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:43.019638 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:43.020316 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:08:43.020351 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:08:43.020658 (+ 307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:08:43.022275 (+ 1617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:43.022278 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:43.022281 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:43.022452 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:43.022675 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:43.022681 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:43.022683 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:43.022726 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:43.023088 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:08:43.023093 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:08:43.023174 (+ 81us) client_negotiation.cc:769] Sending connection context
0903 08:08:43.023268 (+ 94us) client_negotiation.cc:240] Negotiation successful
0903 08:08:43.023401 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":87,"threads_started":1}
I20250903 08:08:43.023844 19914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:43.017397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37140 (local address 127.12.201.254:34143)
0903 08:08:43.017532 (+ 135us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:43.017536 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:43.017741 (+ 205us) server_negotiation.cc:408] Connection header received
0903 08:08:43.017868 (+ 127us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:43.017871 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:43.017915 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:43.017985 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:43.018508 (+ 523us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:43.019038 (+ 530us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:43.019632 (+ 594us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:43.019837 (+ 205us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:43.020774 (+ 937us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:43.020793 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:43.020795 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:08:43.020817 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:43.022171 (+ 1354us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:43.022548 (+ 377us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:43.022551 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:43.022553 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:43.022595 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:43.022803 (+ 208us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:43.022806 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:43.022808 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:43.022942 (+ 134us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:43.023082 (+ 140us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:43.023334 (+ 252us) server_negotiation.cc:300] Negotiation successful
0903 08:08:43.023432 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20250903 08:08:43.024178 19895 tablet_copy_service.cc:140] P ff08db2a004746a6ab7e1d0fa91eed38: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 0fba560828064814bed66061ed3ebc3d ({username='oryx', principal='oryx/127.12.201.252@KRBTEST.COM'} at 127.0.0.1:37140)
I20250903 08:08:43.024274 19895 tablet_copy_service.cc:161] P ff08db2a004746a6ab7e1d0fa91eed38: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 0fba560828064814bed66061ed3ebc3d at {username='oryx', principal='oryx/127.12.201.252@KRBTEST.COM'} at 127.0.0.1:37140: session id = 0fba560828064814bed66061ed3ebc3d-00000000000000000000000000000000
I20250903 08:08:43.025249 19895 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P ff08db2a004746a6ab7e1d0fa91eed38: Tablet Copy: opened 0 blocks and 1 log segments
I20250903 08:08:43.026366 20459 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20250903 08:08:43.028116 20459 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P 0fba560828064814bed66061ed3ebc3d: tablet copy: Starting download of 0 data blocks...
I20250903 08:08:43.028296 20459 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P 0fba560828064814bed66061ed3ebc3d: tablet copy: Starting download of 1 WAL segments...
I20250903 08:08:43.030130 20459 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P 0fba560828064814bed66061ed3ebc3d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250903 08:08:43.031665 19895 tablet_copy_service.cc:342] P ff08db2a004746a6ab7e1d0fa91eed38: Request end of tablet copy session 0fba560828064814bed66061ed3ebc3d-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.12.201.252@KRBTEST.COM'} at 127.0.0.1:37140
I20250903 08:08:43.031741 19895 tablet_copy_service.cc:434] P ff08db2a004746a6ab7e1d0fa91eed38: ending tablet copy session 0fba560828064814bed66061ed3ebc3d-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 0fba560828064814bed66061ed3ebc3d
I20250903 08:08:43.031950 20440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:43.031677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.252:35949 (local address 127.0.0.1:56880)
0903 08:08:43.031820 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:43.031873 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.201.252:35949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20250903 08:08:43.032112 19873 master.cc:679] Network error: unable to get registration information for peer 0fba560828064814bed66061ed3ebc3d: Client connection negotiation failed: client connection to 127.12.201.252:35949: connect: Connection refused (error 111)
W20250903 08:08:43.036078 20459 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20250903 08:08:43.038089 20459 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:08:43.039652 20644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:43.039655 20646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:08:43.040014 20459 server_base.cc:1047] running on GCE node
W20250903 08:08:43.039654 20643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:08:43.040262 20459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:08:43.040555 20459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:08:43.041700 20459 hybrid_clock.cc:648] HybridClock initialized: now 1756886923041687 us; error 31 us; skew 500 ppm
Sep 03 08:08:43 dist-test-slave-27r9 krb5kdc[19827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886923, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.201.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 08:08:43.043346 20459 init.cc:377] Logged in from keytab as oryx/127.12.201.252@KRBTEST.COM (short username oryx)
I20250903 08:08:43.043929 20459 webserver.cc:480] Webserver started at http://127.12.201.252:45569/ using document root <none> and password file <none>
I20250903 08:08:43.044121 20459 fs_manager.cc:362] Metadata directory not provided
I20250903 08:08:43.044158 20459 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:08:43.045292 20459 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20250903 08:08:43.046221 20653 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:43.046434 20459 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250903 08:08:43.046499 20459 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/master-2/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/master-2/wal
uuid: "0fba560828064814bed66061ed3ebc3d"
format_stamp: "Formatted at 2025-09-03 08:08:42 on dist-test-slave-27r9"
server_key: "457b70f2ae351abb894ac9894739b423"
server_key_iv: "ae39ed13e87a046341c973b3a183afa2"
server_key_version: "encryptionkey@0"
I20250903 08:08:43.046594 20459 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:08:43.053339 20459 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:08:43.055249 20459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:08:43.059713 20459 rpc_server.cc:307] RPC server started. Bound to: 127.12.201.252:35949
I20250903 08:08:43.059971 20705 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.201.252:35949 every 8 connection(s)
I20250903 08:08:43.060345 20459 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/master-2/data/info.pb
I20250903 08:08:43.061684 20706 sys_catalog.cc:263] Verifying existing consensus state
I20250903 08:08:43.062456 20706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fba560828064814bed66061ed3ebc3d: Bootstrap starting.
I20250903 08:08:43.072938 20706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fba560828064814bed66061ed3ebc3d: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250903 08:08:43.073258 20706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fba560828064814bed66061ed3ebc3d: Bootstrap complete.
I20250903 08:08:43.074081 20706 raft_consensus.cc:357] T 00000000000000000000000000000000 P 0fba560828064814bed66061ed3ebc3d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ff08db2a004746a6ab7e1d0fa91eed38" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 34143 } } peers { permanent_uuid: "094ea437f04c45fb93ad96244e3ca8a7" member_type: VOTER last_known_addr { host: "127.12.201.253" port: 42205 } attrs { promote: false } } peers { permanent_uuid: "0fba560828064814bed66061ed3ebc3d" member_type: NON_VOTER last_known_addr { host: "127.12.201.252" port: 35949 } attrs { promote: true } }
I20250903 08:08:43.074229 20706 raft_consensus.cc:738] T 00000000000000000000000000000000 P 0fba560828064814bed66061ed3ebc3d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0fba560828064814bed66061ed3ebc3d, State: Initialized, Role: LEARNER
I20250903 08:08:43.074352 20706 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0fba560828064814bed66061ed3ebc3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ff08db2a004746a6ab7e1d0fa91eed38" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 34143 } } peers { permanent_uuid: "094ea437f04c45fb93ad96244e3ca8a7" member_type: VOTER last_known_addr { host: "127.12.201.253" port: 42205 } attrs { promote: false } } peers { permanent_uuid: "0fba560828064814bed66061ed3ebc3d" member_type: NON_VOTER last_known_addr { host: "127.12.201.252" port: 35949 } attrs { promote: true } }
I20250903 08:08:43.074743 20710 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fba560828064814bed66061ed3ebc3d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ff08db2a004746a6ab7e1d0fa91eed38" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 34143 } } peers { permanent_uuid: "094ea437f04c45fb93ad96244e3ca8a7" member_type: VOTER last_known_addr { host: "127.12.201.253" port: 42205 } attrs { promote: false } } peers { permanent_uuid: "0fba560828064814bed66061ed3ebc3d" member_type: NON_VOTER last_known_addr { host: "127.12.201.252" port: 35949 } attrs { promote: true } } }
I20250903 08:08:43.074915 20706 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0fba560828064814bed66061ed3ebc3d [sys.catalog]: configured and running, proceeding with master startup.
I20250903 08:08:43.074939 20710 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fba560828064814bed66061ed3ebc3d [sys.catalog]: This master's current role is: LEARNER
I20250903 08:08:43.076735 20440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:43.067968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.252:35949 (local address 127.0.0.1:56886)
0903 08:08:43.068216 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:43.068233 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:43.068332 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:43.069021 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:43.069025 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:43.069228 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:08:43.069479 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:43.069488 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:43.070823 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:43.070826 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:43.071366 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:43.071377 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:43.071493 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:43.072099 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:08:43.072118 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:08:43.072551 (+ 433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:08:43.075042 (+ 2491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:43.075047 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:43.075049 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:43.075286 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:43.075580 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:43.075586 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:43.075590 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:43.075655 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:43.076128 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:08:43.076133 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:08:43.076251 (+ 118us) client_negotiation.cc:769] Sending connection context
0903 08:08:43.076388 (+ 137us) client_negotiation.cc:240] Negotiation successful
0903 08:08:43.076546 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164}
I20250903 08:08:43.077564 20709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:43.068477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56886 (local address 127.12.201.252:35949)
0903 08:08:43.068760 (+ 283us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:43.068764 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:43.068779 (+ 15us) server_negotiation.cc:408] Connection header received
0903 08:08:43.068825 (+ 46us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:43.068828 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:43.068880 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:43.068982 (+ 102us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:43.069617 (+ 635us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:43.070689 (+ 1072us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:43.071733 (+ 1044us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:43.072012 (+ 279us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:43.072755 (+ 743us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:43.072771 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:43.072774 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:08:43.072796 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:43.074865 (+ 2069us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:43.075425 (+ 560us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:43.075429 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:43.075431 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:43.075477 (+ 46us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:43.075776 (+ 299us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:43.075783 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:43.075787 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:43.075986 (+ 199us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:43.077054 (+ 1068us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:43.077140 (+ 86us) server_negotiation.cc:300] Negotiation successful
0903 08:08:43.077280 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":102,"threads_started":1}
I20250903 08:08:43.080667 20721 catalog_manager.cc:1261] Loaded cluster ID: b7b764f24fd443ee92ab54013d2600ab
I20250903 08:08:43.080740 20721 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 0fba560828064814bed66061ed3ebc3d: loading cluster ID for follower catalog manager: success
I20250903 08:08:43.083212 20721 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 0fba560828064814bed66061ed3ebc3d: acquiring CA information for follower catalog manager: success
I20250903 08:08:43.084766 20721 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 0fba560828064814bed66061ed3ebc3d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250903 08:08:43.161515 20204 heartbeater.cc:499] Master 127.12.201.254:34143 was elected leader, sending a full tablet report...
I20250903 08:08:43.382727 20340 heartbeater.cc:499] Master 127.12.201.254:34143 was elected leader, sending a full tablet report...
I20250903 08:08:43.440744 20680 raft_consensus.cc:1215] T 00000000000000000000000000000000 P 0fba560828064814bed66061ed3ebc3d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20250903 08:08:43.441149 20710 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fba560828064814bed66061ed3ebc3d [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff08db2a004746a6ab7e1d0fa91eed38. Latest consensus state: current_term: 1 leader_uuid: "ff08db2a004746a6ab7e1d0fa91eed38" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ff08db2a004746a6ab7e1d0fa91eed38" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 34143 } } peers { permanent_uuid: "094ea437f04c45fb93ad96244e3ca8a7" member_type: VOTER last_known_addr { host: "127.12.201.253" port: 42205 } attrs { promote: false } } peers { permanent_uuid: "0fba560828064814bed66061ed3ebc3d" member_type: NON_VOTER last_known_addr { host: "127.12.201.252" port: 35949 } attrs { promote: true } } }
I20250903 08:08:43.441272 20710 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fba560828064814bed66061ed3ebc3d [sys.catalog]: This master's current role is: LEARNER
I20250903 08:08:43.982493 20445 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff08db2a004746a6ab7e1d0fa91eed38 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ff08db2a004746a6ab7e1d0fa91eed38" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ff08db2a004746a6ab7e1d0fa91eed38" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 34143 } } peers { permanent_uuid: "094ea437f04c45fb93ad96244e3ca8a7" member_type: VOTER last_known_addr { host: "127.12.201.253" port: 42205 } attrs { promote: false } } peers { permanent_uuid: "0fba560828064814bed66061ed3ebc3d" member_type: NON_VOTER last_known_addr { host: "127.12.201.252" port: 35949 } attrs { promote: true } } }
I20250903 08:08:43.982692 20445 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff08db2a004746a6ab7e1d0fa91eed38 [sys.catalog]: This master's current role is: LEADER
I20250903 08:08:43.983620 20723 raft_consensus.cc:1062] T 00000000000000000000000000000000 P ff08db2a004746a6ab7e1d0fa91eed38: attempting to promote NON_VOTER 0fba560828064814bed66061ed3ebc3d to VOTER
I20250903 08:08:43.984009 20723 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff08db2a004746a6ab7e1d0fa91eed38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ff08db2a004746a6ab7e1d0fa91eed38" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 34143 } } peers { permanent_uuid: "094ea437f04c45fb93ad96244e3ca8a7" member_type: VOTER last_known_addr { host: "127.12.201.253" port: 42205 } attrs { promote: false } } peers { permanent_uuid: "0fba560828064814bed66061ed3ebc3d" member_type: VOTER last_known_addr { host: "127.12.201.252" port: 35949 } attrs { promote: false } }
I20250903 08:08:43.985181 20680 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 0fba560828064814bed66061ed3ebc3d [term 1 LEARNER]: Refusing update from remote peer ff08db2a004746a6ab7e1d0fa91eed38: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250903 08:08:43.985234 20497 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 094ea437f04c45fb93ad96244e3ca8a7 [term 1 FOLLOWER]: Refusing update from remote peer ff08db2a004746a6ab7e1d0fa91eed38: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250903 08:08:43.985577 20722 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ff08db2a004746a6ab7e1d0fa91eed38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "094ea437f04c45fb93ad96244e3ca8a7" member_type: VOTER last_known_addr { host: "127.12.201.253" port: 42205 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250903 08:08:43.985787 20723 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ff08db2a004746a6ab7e1d0fa91eed38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fba560828064814bed66061ed3ebc3d" member_type: VOTER last_known_addr { host: "127.12.201.252" port: 35949 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250903 08:08:43.987499 20723 raft_consensus.cc:2953] T 00000000000000000000000000000000 P ff08db2a004746a6ab7e1d0fa91eed38 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0fba560828064814bed66061ed3ebc3d (127.12.201.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ff08db2a004746a6ab7e1d0fa91eed38" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 34143 } } peers { permanent_uuid: "094ea437f04c45fb93ad96244e3ca8a7" member_type: VOTER last_known_addr { host: "127.12.201.253" port: 42205 } attrs { promote: false } } peers { permanent_uuid: "0fba560828064814bed66061ed3ebc3d" member_type: VOTER last_known_addr { host: "127.12.201.252" port: 35949 } attrs { promote: false } } }
I20250903 08:08:43.987990 20497 raft_consensus.cc:2953] T 00000000000000000000000000000000 P 094ea437f04c45fb93ad96244e3ca8a7 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0fba560828064814bed66061ed3ebc3d (127.12.201.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ff08db2a004746a6ab7e1d0fa91eed38" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 34143 } } peers { permanent_uuid: "094ea437f04c45fb93ad96244e3ca8a7" member_type: VOTER last_known_addr { host: "127.12.201.253" port: 42205 } attrs { promote: false } } peers { permanent_uuid: "0fba560828064814bed66061ed3ebc3d" member_type: VOTER last_known_addr { host: "127.12.201.252" port: 35949 } attrs { promote: false } } }
I20250903 08:08:43.989130 20722 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff08db2a004746a6ab7e1d0fa91eed38 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ff08db2a004746a6ab7e1d0fa91eed38" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ff08db2a004746a6ab7e1d0fa91eed38" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 34143 } } peers { permanent_uuid: "094ea437f04c45fb93ad96244e3ca8a7" member_type: VOTER last_known_addr { host: "127.12.201.253" port: 42205 } attrs { promote: false } } peers { permanent_uuid: "0fba560828064814bed66061ed3ebc3d" member_type: VOTER last_known_addr { host: "127.12.201.252" port: 35949 } attrs { promote: false } } }
I20250903 08:08:43.989216 20722 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff08db2a004746a6ab7e1d0fa91eed38 [sys.catalog]: This master's current role is: LEADER
I20250903 08:08:43.989178 20723 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff08db2a004746a6ab7e1d0fa91eed38 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ff08db2a004746a6ab7e1d0fa91eed38" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ff08db2a004746a6ab7e1d0fa91eed38" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 34143 } } peers { permanent_uuid: "094ea437f04c45fb93ad96244e3ca8a7" member_type: VOTER last_known_addr { host: "127.12.201.253" port: 42205 } attrs { promote: false } } peers { permanent_uuid: "0fba560828064814bed66061ed3ebc3d" member_type: VOTER last_known_addr { host: "127.12.201.252" port: 35949 } attrs { promote: false } } }
I20250903 08:08:43.989274 20723 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff08db2a004746a6ab7e1d0fa91eed38 [sys.catalog]: This master's current role is: LEADER
I20250903 08:08:43.989003 20680 raft_consensus.cc:2953] T 00000000000000000000000000000000 P 0fba560828064814bed66061ed3ebc3d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0fba560828064814bed66061ed3ebc3d (127.12.201.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ff08db2a004746a6ab7e1d0fa91eed38" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 34143 } } peers { permanent_uuid: "094ea437f04c45fb93ad96244e3ca8a7" member_type: VOTER last_known_addr { host: "127.12.201.253" port: 42205 } attrs { promote: false } } peers { permanent_uuid: "0fba560828064814bed66061ed3ebc3d" member_type: VOTER last_known_addr { host: "127.12.201.252" port: 35949 } attrs { promote: false } } }
I20250903 08:08:43.989326 20722 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff08db2a004746a6ab7e1d0fa91eed38 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ff08db2a004746a6ab7e1d0fa91eed38" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ff08db2a004746a6ab7e1d0fa91eed38" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 34143 } } peers { permanent_uuid: "094ea437f04c45fb93ad96244e3ca8a7" member_type: VOTER last_known_addr { host: "127.12.201.253" port: 42205 } attrs { promote: false } } peers { permanent_uuid: "0fba560828064814bed66061ed3ebc3d" member_type: VOTER last_known_addr { host: "127.12.201.252" port: 35949 } attrs { promote: false } } }
I20250903 08:08:43.989399 20722 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff08db2a004746a6ab7e1d0fa91eed38 [sys.catalog]: This master's current role is: LEADER
I20250903 08:08:43.989755 20726 sys_catalog.cc:455] T 00000000000000000000000000000000 P 094ea437f04c45fb93ad96244e3ca8a7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ff08db2a004746a6ab7e1d0fa91eed38" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ff08db2a004746a6ab7e1d0fa91eed38" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 34143 } } peers { permanent_uuid: "094ea437f04c45fb93ad96244e3ca8a7" member_type: VOTER last_known_addr { host: "127.12.201.253" port: 42205 } attrs { promote: false } } peers { permanent_uuid: "0fba560828064814bed66061ed3ebc3d" member_type: VOTER last_known_addr { host: "127.12.201.252" port: 35949 } attrs { promote: false } } }
I20250903 08:08:43.989840 20726 sys_catalog.cc:458] T 00000000000000000000000000000000 P 094ea437f04c45fb93ad96244e3ca8a7 [sys.catalog]: This master's current role is: FOLLOWER
I20250903 08:08:43.990599 20727 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fba560828064814bed66061ed3ebc3d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ff08db2a004746a6ab7e1d0fa91eed38" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ff08db2a004746a6ab7e1d0fa91eed38" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 34143 } } peers { permanent_uuid: "094ea437f04c45fb93ad96244e3ca8a7" member_type: VOTER last_known_addr { host: "127.12.201.253" port: 42205 } attrs { promote: false } } peers { permanent_uuid: "0fba560828064814bed66061ed3ebc3d" member_type: VOTER last_known_addr { host: "127.12.201.252" port: 35949 } attrs { promote: false } } }
I20250903 08:08:43.990722 20727 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fba560828064814bed66061ed3ebc3d [sys.catalog]: This master's current role is: FOLLOWER
I20250903 08:08:44.067467 20729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:44.057971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38744 (local address 127.12.201.253:42205)
0903 08:08:44.058212 (+ 241us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:44.058216 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:44.059247 (+ 1031us) server_negotiation.cc:408] Connection header received
0903 08:08:44.059370 (+ 123us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:44.059373 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:44.059420 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:44.059499 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:44.060370 (+ 871us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.061149 (+ 779us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:44.062223 (+ 1074us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.062425 (+ 202us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:44.063668 (+ 1243us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:44.063693 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:44.063697 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:08:44.063733 (+ 36us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:44.065888 (+ 2155us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:44.066377 (+ 489us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:44.066381 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:44.066384 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:44.066439 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:44.066678 (+ 239us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:44.066680 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:44.066681 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:44.066828 (+ 147us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:44.066923 (+ 95us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:44.067179 (+ 256us) server_negotiation.cc:300] Negotiation successful
0903 08:08:44.067283 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":85,"threads_started":1}
I20250903 08:08:44.071581 20730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:44.068220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:34143 (local address 127.0.0.1:37156)
0903 08:08:44.068447 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:44.068465 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:44.068533 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:44.069806 (+ 1273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:44.069809 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:44.069822 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 08:08:44.069991 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:44.069995 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.070683 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:44.070685 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:44.071281 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:44.071288 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.071377 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:44.071392 (+ 15us) client_negotiation.cc:769] Sending connection context
0903 08:08:44.071428 (+ 36us) client_negotiation.cc:240] Negotiation successful
0903 08:08:44.071480 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":82,"threads_started":1}
I20250903 08:08:44.071980 20731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:44.068326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37156 (local address 127.12.201.254:34143)
0903 08:08:44.069575 (+ 1249us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:44.069590 (+ 15us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:44.069604 (+ 14us) server_negotiation.cc:408] Connection header received
0903 08:08:44.069664 (+ 60us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:44.069666 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:44.069709 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:44.069794 (+ 85us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 08:08:44.070091 (+ 297us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.070605 (+ 514us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:44.071369 (+ 764us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.071738 (+ 369us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:44.071773 (+ 35us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:44.071823 (+ 50us) server_negotiation.cc:300] Negotiation successful
0903 08:08:44.071879 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1175,"thread_start_us":97,"threads_started":1}
I20250903 08:08:44.075238 20730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:44.073103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.253:42205 (local address 127.0.0.1:38746)
0903 08:08:44.073211 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:44.073225 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:44.073306 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:44.073514 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:44.073516 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:44.073532 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 08:08:44.073680 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:44.073685 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.074314 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:44.074317 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:44.074862 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:44.074869 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.074973 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:44.075014 (+ 41us) client_negotiation.cc:769] Sending connection context
0903 08:08:44.075068 (+ 54us) client_negotiation.cc:240] Negotiation successful
0903 08:08:44.075118 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20250903 08:08:44.075620 20729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:44.073133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38746 (local address 127.12.201.253:42205)
0903 08:08:44.073219 (+ 86us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:44.073225 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:44.073278 (+ 53us) server_negotiation.cc:408] Connection header received
0903 08:08:44.073396 (+ 118us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:44.073399 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:44.073436 (+ 37us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:44.073504 (+ 68us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 08:08:44.073772 (+ 268us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.074221 (+ 449us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:44.074965 (+ 744us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.075431 (+ 466us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:44.075465 (+ 34us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:44.075510 (+ 45us) server_negotiation.cc:300] Negotiation successful
0903 08:08:44.075552 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20250903 08:08:44.079151 20730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:44.076681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.252:35949 (local address 127.0.0.1:56900)
0903 08:08:44.076779 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:44.076791 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:44.076848 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:44.077254 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:44.077257 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:44.077273 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 08:08:44.077414 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:44.077419 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.078188 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:44.078192 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:44.078778 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:44.078786 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.078892 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:44.078909 (+ 17us) client_negotiation.cc:769] Sending connection context
0903 08:08:44.078959 (+ 50us) client_negotiation.cc:240] Negotiation successful
0903 08:08:44.079038 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20250903 08:08:44.079751 20732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:44.076816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56900 (local address 127.12.201.252:35949)
0903 08:08:44.077038 (+ 222us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:44.077041 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:44.077051 (+ 10us) server_negotiation.cc:408] Connection header received
0903 08:08:44.077090 (+ 39us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:44.077092 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:44.077140 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:44.077236 (+ 96us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 08:08:44.077526 (+ 290us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.078086 (+ 560us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:44.078891 (+ 805us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.079365 (+ 474us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:44.079482 (+ 117us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:44.079549 (+ 67us) server_negotiation.cc:300] Negotiation successful
0903 08:08:44.079614 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":67,"threads_started":1}
I20250903 08:08:44.087461 20732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:44.081460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56914 (local address 127.12.201.252:35949)
0903 08:08:44.081561 (+ 101us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:44.081564 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:44.081591 (+ 27us) server_negotiation.cc:408] Connection header received
0903 08:08:44.081698 (+ 107us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:44.081700 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:44.081744 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:44.081806 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:44.082511 (+ 705us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.082963 (+ 452us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:44.083569 (+ 606us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.083688 (+ 119us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:44.084731 (+ 1043us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:44.084746 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:44.084748 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:08:44.084768 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:44.086235 (+ 1467us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:44.086580 (+ 345us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:44.086582 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:44.086584 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:44.086616 (+ 32us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:44.086840 (+ 224us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:44.086842 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:44.086843 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:44.086962 (+ 119us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:44.087101 (+ 139us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:44.087257 (+ 156us) server_negotiation.cc:300] Negotiation successful
0903 08:08:44.087347 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20250903 08:08:44.090310 20733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:44.088173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:34143 (local address 127.0.0.1:37162)
0903 08:08:44.088405 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:44.088416 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:44.088491 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:44.088671 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:44.088672 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:44.088684 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 08:08:44.088858 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:44.088862 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.089486 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:44.089488 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:44.090040 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:44.090046 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.090128 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:44.090141 (+ 13us) client_negotiation.cc:769] Sending connection context
0903 08:08:44.090176 (+ 35us) client_negotiation.cc:240] Negotiation successful
0903 08:08:44.090215 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":86,"threads_started":1}
I20250903 08:08:44.090713 20731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:44.088300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37162 (local address 127.12.201.254:34143)
0903 08:08:44.088416 (+ 116us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:44.088419 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:44.088459 (+ 40us) server_negotiation.cc:408] Connection header received
0903 08:08:44.088556 (+ 97us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:44.088559 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:44.088599 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:44.088662 (+ 63us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 08:08:44.088934 (+ 272us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.089406 (+ 472us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:44.090120 (+ 714us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.090485 (+ 365us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:44.090519 (+ 34us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:44.090570 (+ 51us) server_negotiation.cc:300] Negotiation successful
0903 08:08:44.090623 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20250903 08:08:44.093923 20733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:44.091783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.253:42205 (local address 127.0.0.1:38754)
0903 08:08:44.091894 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:44.091905 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:44.091967 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:44.092180 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:44.092183 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:44.092196 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 08:08:44.092339 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:44.092343 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.092988 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:44.092991 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:44.093623 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:44.093636 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.093732 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:44.093745 (+ 13us) client_negotiation.cc:769] Sending connection context
0903 08:08:44.093790 (+ 45us) client_negotiation.cc:240] Negotiation successful
0903 08:08:44.093830 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20250903 08:08:44.094262 20729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:44.091893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38754 (local address 127.12.201.253:42205)
0903 08:08:44.092011 (+ 118us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:44.092014 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:44.092026 (+ 12us) server_negotiation.cc:408] Connection header received
0903 08:08:44.092066 (+ 40us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:44.092068 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:44.092108 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:44.092169 (+ 61us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 08:08:44.092418 (+ 249us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.092875 (+ 457us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:44.093725 (+ 850us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.094027 (+ 302us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:44.094067 (+ 40us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:44.094118 (+ 51us) server_negotiation.cc:300] Negotiation successful
0903 08:08:44.094178 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20250903 08:08:44.097430 20733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:44.095320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.252:35949 (local address 127.0.0.1:56924)
0903 08:08:44.095461 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:44.095475 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:44.095560 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:44.095756 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:44.095759 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:44.095772 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 08:08:44.095920 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:44.095924 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.096558 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:44.096561 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:44.097088 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:44.097094 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.097193 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:44.097209 (+ 16us) client_negotiation.cc:769] Sending connection context
0903 08:08:44.097258 (+ 49us) client_negotiation.cc:240] Negotiation successful
0903 08:08:44.097307 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20250903 08:08:44.097782 20732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:44.095384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56924 (local address 127.12.201.252:35949)
0903 08:08:44.095469 (+ 85us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:44.095475 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:44.095519 (+ 44us) server_negotiation.cc:408] Connection header received
0903 08:08:44.095646 (+ 127us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:44.095649 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:44.095684 (+ 35us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:44.095744 (+ 60us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 08:08:44.096008 (+ 264us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.096464 (+ 456us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:44.097188 (+ 724us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.097620 (+ 432us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:44.097646 (+ 26us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:44.097681 (+ 35us) server_negotiation.cc:300] Negotiation successful
0903 08:08:44.097717 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20250903 08:08:44.099064 13095 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUML90r/build/debug/bin/kudu with pid 19939
I20250903 08:08:44.105662 13095 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUML90r/build/debug/bin/kudu
/tmp/dist-test-taskUML90r/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.201.193:43289
--local_ip_for_outbound_sockets=127.12.201.193
--tserver_master_addrs=127.12.201.254:34143
--webserver_port=44727
--webserver_interface=127.12.201.193
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/krb5kdc/oryx.keytab
--principal=oryx/127.12.201.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.201.212:37903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.12.201.254:34143,127.12.201.253:42205
--tserver_master_addrs=127.12.201.254:34143,127.12.201.253:42205,127.12.201.252:35949 with env {KRB5CCNAME=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 08:08:44.210510 20734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:08:44.210736 20734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:08:44.210817 20734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:08:44.214025 20734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 08:08:44.214089 20734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:08:44.214201 20734 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.201.193
I20250903 08:08:44.218253 20734 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.201.212:37903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/krb5kdc/oryx.keytab
--principal=oryx/127.12.201.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.201.193:43289
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.201.193
--webserver_port=44727
--webserver_require_spnego=true
--tserver_master_addrs=127.12.201.254:34143,127.12.201.253:42205,127.12.201.252:35949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20734
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.201.193
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 07:50:08 UTC on 5fd53c4cbb9d
build id 7874
I20250903 08:08:44.219331 20734 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:08:44.220151 20734 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:08:44.226303 20742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:44.226312 20740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:44.226303 20739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:08:44.226584 20734 server_base.cc:1047] running on GCE node
I20250903 08:08:44.226910 20734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:08:44.227456 20734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:08:44.228605 20734 hybrid_clock.cc:648] HybridClock initialized: now 1756886924228594 us; error 24 us; skew 500 ppm
Sep 03 08:08:44 dist-test-slave-27r9 krb5kdc[19827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886924, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.201.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 08:08:44.231231 20734 init.cc:377] Logged in from keytab as oryx/127.12.201.193@KRBTEST.COM (short username oryx)
I20250903 08:08:44.232240 20734 webserver.cc:480] Webserver started at http://127.12.201.193:44727/ using document root <none> and password file <none>
I20250903 08:08:44.232772 20734 fs_manager.cc:362] Metadata directory not provided
I20250903 08:08:44.232831 20734 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:08:44.236318 20734 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20250903 08:08:44.238257 20749 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:44.239310 20734 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250903 08:08:44.239423 20734 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-0/wal
uuid: "a16339bb0a5e4d3e93e3083c087b5fe7"
format_stamp: "Formatted at 2025-09-03 08:08:41 on dist-test-slave-27r9"
server_key: "0093463df70c666b37d84b3104151f8f"
server_key_iv: "46cf715b43c363fafe47dc28353a4411"
server_key_version: "encryptionkey@0"
I20250903 08:08:44.239852 20734 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:08:44.265980 20734 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:08:44.268851 20734 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:08:44.269035 20734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:08:44.269609 20734 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 08:08:44.270489 20734 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 08:08:44.270550 20734 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:44.270613 20734 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 08:08:44.270653 20734 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:44.280292 20734 rpc_server.cc:307] RPC server started. Bound to: 127.12.201.193:43289
I20250903 08:08:44.280315 20862 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.201.193:43289 every 8 connection(s)
I20250903 08:08:44.281381 20734 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-0/data/info.pb
Sep 03 08:08:44 dist-test-slave-27r9 krb5kdc[19827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886924, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.201.193@KRBTEST.COM for oryx/127.12.201.252@KRBTEST.COM
Sep 03 08:08:44 dist-test-slave-27r9 krb5kdc[19827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886924, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.201.193@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM
I20250903 08:08:44.291549 13095 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUML90r/build/debug/bin/kudu as pid 20734
Sep 03 08:08:44 dist-test-slave-27r9 krb5kdc[19827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886924, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.201.193@KRBTEST.COM for oryx/127.12.201.253@KRBTEST.COM
I20250903 08:08:44.291770 13095 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUML90r/build/debug/bin/kudu with pid 20075
I20250903 08:08:44.295221 20732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:44.283105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.193:53271 (local address 127.12.201.252:35949)
0903 08:08:44.283263 (+ 158us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:44.283268 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:44.284000 (+ 732us) server_negotiation.cc:408] Connection header received
0903 08:08:44.285100 (+ 1100us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:44.285104 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:44.285157 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:44.285209 (+ 52us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:44.286660 (+ 1451us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.287426 (+ 766us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:44.288049 (+ 623us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.288222 (+ 173us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:44.290555 (+ 2333us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:44.290575 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:44.290579 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:08:44.290610 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:44.292561 (+ 1951us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:44.293459 (+ 898us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:44.293462 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:44.293464 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:44.293521 (+ 57us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:44.293878 (+ 357us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:44.293884 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:44.293887 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:44.294085 (+ 198us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:44.294162 (+ 77us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:44.294866 (+ 704us) server_negotiation.cc:300] Negotiation successful
0903 08:08:44.295025 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20250903 08:08:44.295423 20731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:44.283187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.193:49167 (local address 127.12.201.254:34143)
0903 08:08:44.283316 (+ 129us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:44.283320 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:44.284007 (+ 687us) server_negotiation.cc:408] Connection header received
0903 08:08:44.285078 (+ 1071us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:44.285087 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:44.285149 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:44.285228 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:44.286603 (+ 1375us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.287419 (+ 816us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:44.288148 (+ 729us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.288333 (+ 185us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:44.290796 (+ 2463us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:44.290814 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:44.290816 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:08:44.290841 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:44.293529 (+ 2688us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:44.294018 (+ 489us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:44.294025 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:44.294030 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:44.294084 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:44.294359 (+ 275us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:44.294365 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:44.294370 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:44.294574 (+ 204us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:44.294650 (+ 76us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:44.295036 (+ 386us) server_negotiation.cc:300] Negotiation successful
0903 08:08:44.295181 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20250903 08:08:44.296048 20729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:44.284478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.193:47749 (local address 127.12.201.253:42205)
0903 08:08:44.284609 (+ 131us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:44.284612 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:44.284708 (+ 96us) server_negotiation.cc:408] Connection header received
0903 08:08:44.285101 (+ 393us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:44.285104 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:44.285157 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:44.285209 (+ 52us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:44.286648 (+ 1439us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.287262 (+ 614us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:44.288103 (+ 841us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.288286 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:44.292323 (+ 4037us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:44.292345 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:44.292350 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:08:44.292382 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:44.294386 (+ 2004us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:44.294814 (+ 428us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:44.294818 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:44.294819 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:44.294866 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:44.295159 (+ 293us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:44.295163 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:44.295165 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:44.295361 (+ 196us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:44.295430 (+ 69us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:44.295782 (+ 352us) server_negotiation.cc:300] Negotiation successful
0903 08:08:44.295900 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20250903 08:08:44.296805 20869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:44.284354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.253:42205 (local address 127.12.201.193:47749)
0903 08:08:44.284614 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:44.284627 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:44.284898 (+ 271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:44.285393 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:44.285402 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:44.285915 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:08:44.286500 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:44.286509 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.287377 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:44.287380 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:44.287969 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:44.287977 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.288403 (+ 426us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:44.289190 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:08:44.289204 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:08:44.292170 (+ 2966us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:08:44.294507 (+ 2337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:44.294510 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:44.294512 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:44.294698 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:44.295004 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:44.295008 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:44.295010 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:44.295059 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:44.295507 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:08:44.295510 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:08:44.295590 (+ 80us) client_negotiation.cc:769] Sending connection context
0903 08:08:44.296167 (+ 577us) client_negotiation.cc:240] Negotiation successful
0903 08:08:44.296294 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"spinlock_wait_cycles":5888,"thread_start_us":100,"threads_started":1}
I20250903 08:08:44.296803 20868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:44.283413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:34143 (local address 127.12.201.193:49167)
0903 08:08:44.283866 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:44.283910 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:44.284893 (+ 983us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:44.285392 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:44.285400 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:44.285898 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:08:44.286457 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:44.286470 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.287525 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:44.287528 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:44.288026 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:44.288036 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.288428 (+ 392us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:44.289080 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:08:44.289092 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:08:44.290682 (+ 1590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:08:44.293660 (+ 2978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:44.293667 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:44.293672 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:44.293904 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:44.294191 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:44.294196 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:44.294201 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:44.294261 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:44.294722 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:08:44.294728 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:08:44.294818 (+ 90us) client_negotiation.cc:769] Sending connection context
0903 08:08:44.295596 (+ 778us) client_negotiation.cc:240] Negotiation successful
0903 08:08:44.295899 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"mutex_wait_us":21,"spinlock_wait_cycles":41984,"thread_start_us":136,"threads_started":1}
I20250903 08:08:44.297590 20867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:44.283413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.252:35949 (local address 127.12.201.193:53271)
0903 08:08:44.283866 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:44.283910 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:44.284890 (+ 980us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:44.285393 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:44.285402 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:44.286315 (+ 913us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:08:44.286565 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:44.286574 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.287563 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:44.287566 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:44.287931 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:44.287938 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.288126 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:44.288765 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:08:44.288803 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:08:44.290385 (+ 1582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:08:44.293044 (+ 2659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:44.293050 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:44.293061 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:44.293353 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:44.293602 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:44.293607 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:44.293611 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:44.293740 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:44.294250 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:08:44.294259 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:08:44.294558 (+ 299us) client_negotiation.cc:769] Sending connection context
0903 08:08:44.297146 (+ 2588us) client_negotiation.cc:240] Negotiation successful
0903 08:08:44.297472 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"spinlock_wait_cycles":9216,"thread_start_us":116,"threads_started":1}
I20250903 08:08:44.298053 20865 heartbeater.cc:344] Connected to a master server at 127.12.201.253:42205
I20250903 08:08:44.298280 20865 heartbeater.cc:461] Registering TS with master...
I20250903 08:08:44.298636 20863 heartbeater.cc:344] Connected to a master server at 127.12.201.252:35949
I20250903 08:08:44.298715 20863 heartbeater.cc:461] Registering TS with master...
I20250903 08:08:44.298815 20865 heartbeater.cc:507] Master 127.12.201.253:42205 requested a full tablet report, sending...
I20250903 08:08:44.298942 20863 heartbeater.cc:507] Master 127.12.201.252:35949 requested a full tablet report, sending...
I20250903 08:08:44.299108 20864 heartbeater.cc:344] Connected to a master server at 127.12.201.254:34143
I20250903 08:08:44.299172 20864 heartbeater.cc:461] Registering TS with master...
I20250903 08:08:44.299341 20864 heartbeater.cc:507] Master 127.12.201.254:34143 requested a full tablet report, sending...
I20250903 08:08:44.300031 19873 ts_manager.cc:194] Re-registered known tserver with Master: a16339bb0a5e4d3e93e3083c087b5fe7 (127.12.201.193:43289)
I20250903 08:08:44.300226 20670 ts_manager.cc:194] Registered new tserver with Master: a16339bb0a5e4d3e93e3083c087b5fe7 (127.12.201.193:43289)
I20250903 08:08:44.300231 20486 ts_manager.cc:194] Registered new tserver with Master: a16339bb0a5e4d3e93e3083c087b5fe7 (127.12.201.193:43289)
I20250903 08:08:44.300715 19873 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.201.193@KRBTEST.COM'} at 127.12.201.193:49167
I20250903 08:08:44.302699 13095 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUML90r/build/debug/bin/kudu
/tmp/dist-test-taskUML90r/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.201.194:41585
--local_ip_for_outbound_sockets=127.12.201.194
--tserver_master_addrs=127.12.201.254:34143
--webserver_port=37469
--webserver_interface=127.12.201.194
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/krb5kdc/oryx.keytab
--principal=oryx/127.12.201.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.201.212:37903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.12.201.254:34143,127.12.201.253:42205
--tserver_master_addrs=127.12.201.254:34143,127.12.201.253:42205,127.12.201.252:35949 with env {KRB5CCNAME=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 08:08:44.409313 20870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:08:44.409538 20870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:08:44.409612 20870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:08:44.412853 20870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 08:08:44.412925 20870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:08:44.413033 20870 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.201.194
I20250903 08:08:44.417213 20870 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.201.212:37903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/krb5kdc/oryx.keytab
--principal=oryx/127.12.201.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.201.194:41585
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.201.194
--webserver_port=37469
--webserver_require_spnego=true
--tserver_master_addrs=127.12.201.254:34143,127.12.201.253:42205,127.12.201.252:35949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20870
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.201.194
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 07:50:08 UTC on 5fd53c4cbb9d
build id 7874
I20250903 08:08:44.418282 20870 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:08:44.419157 20870 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:08:44.426298 20875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:44.426298 20876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:44.426332 20878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:08:44.426659 20870 server_base.cc:1047] running on GCE node
I20250903 08:08:44.427098 20870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:08:44.427721 20870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:08:44.428896 20870 hybrid_clock.cc:648] HybridClock initialized: now 1756886924428867 us; error 42 us; skew 500 ppm
Sep 03 08:08:44 dist-test-slave-27r9 krb5kdc[19827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886924, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.201.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 08:08:44.431584 20870 init.cc:377] Logged in from keytab as oryx/127.12.201.194@KRBTEST.COM (short username oryx)
I20250903 08:08:44.432648 20870 webserver.cc:480] Webserver started at http://127.12.201.194:37469/ using document root <none> and password file <none>
I20250903 08:08:44.433198 20870 fs_manager.cc:362] Metadata directory not provided
I20250903 08:08:44.433257 20870 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:08:44.436743 20870 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20250903 08:08:44.438657 20885 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:44.439666 20870 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250903 08:08:44.439790 20870 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-1/wal
uuid: "2b6d679b81794eac94e52a9454286dcd"
format_stamp: "Formatted at 2025-09-03 08:08:42 on dist-test-slave-27r9"
server_key: "67df2e7f0e479cd54c1ac8a5df30fd44"
server_key_iv: "51463c2674076fa112f7ed5bb9ad5df0"
server_key_version: "encryptionkey@0"
I20250903 08:08:44.440208 20870 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:08:44.454092 20870 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:08:44.457160 20870 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:08:44.457353 20870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:08:44.457912 20870 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 08:08:44.458772 20870 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 08:08:44.458817 20870 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:44.458866 20870 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 08:08:44.458897 20870 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:44.467890 20870 rpc_server.cc:307] RPC server started. Bound to: 127.12.201.194:41585
I20250903 08:08:44.467904 20998 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.201.194:41585 every 8 connection(s)
I20250903 08:08:44.469137 20870 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-1/data/info.pb
Sep 03 08:08:44 dist-test-slave-27r9 krb5kdc[19827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886924, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.201.194@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM
I20250903 08:08:44.478068 13095 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUML90r/build/debug/bin/kudu as pid 20870
I20250903 08:08:44.478240 13095 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUML90r/build/debug/bin/kudu with pid 20211
Sep 03 08:08:44 dist-test-slave-27r9 krb5kdc[19827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886924, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.201.194@KRBTEST.COM for oryx/127.12.201.253@KRBTEST.COM
Sep 03 08:08:44 dist-test-slave-27r9 krb5kdc[19827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886924, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.201.194@KRBTEST.COM for oryx/127.12.201.252@KRBTEST.COM
I20250903 08:08:44.482391 20731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:44.470896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.194:51679 (local address 127.12.201.254:34143)
0903 08:08:44.471037 (+ 141us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:44.471041 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:44.471747 (+ 706us) server_negotiation.cc:408] Connection header received
0903 08:08:44.472907 (+ 1160us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:44.472910 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:44.472967 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:44.473016 (+ 49us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:44.474481 (+ 1465us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.475263 (+ 782us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:44.475863 (+ 600us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.476049 (+ 186us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:44.478628 (+ 2579us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:44.478649 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:44.478652 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:08:44.478684 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:44.480289 (+ 1605us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:44.480905 (+ 616us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:44.480909 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:44.480911 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:44.480968 (+ 57us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:44.481302 (+ 334us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:44.481309 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:44.481313 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:44.481481 (+ 168us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:44.481587 (+ 106us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:44.482080 (+ 493us) server_negotiation.cc:300] Negotiation successful
0903 08:08:44.482212 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20250903 08:08:44.483180 21003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:44.471151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:34143 (local address 127.12.201.194:51679)
0903 08:08:44.471593 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:44.471636 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:44.472689 (+ 1053us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:44.473174 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:44.473181 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:44.473734 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:08:44.474306 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:44.474317 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.475385 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:44.475388 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:44.475754 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:44.475762 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.476004 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:44.476617 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:08:44.476739 (+ 122us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:08:44.478432 (+ 1693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:08:44.480405 (+ 1973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:44.480414 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:44.480430 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:44.480792 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:44.481066 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:44.481072 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:44.481077 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:44.481201 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:44.481596 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:08:44.481606 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:08:44.481842 (+ 236us) client_negotiation.cc:769] Sending connection context
0903 08:08:44.482034 (+ 192us) client_negotiation.cc:240] Negotiation successful
0903 08:08:44.482317 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"spinlock_wait_cycles":3328,"thread_start_us":108,"threads_started":1}
I20250903 08:08:44.484112 21005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:44.471151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.252:35949 (local address 127.12.201.194:59201)
0903 08:08:44.472225 (+ 1074us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:44.472240 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:44.472693 (+ 453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:44.473171 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:44.473179 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:44.473712 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:08:44.474298 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:44.474310 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.475326 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:44.475329 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:44.475829 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:44.475835 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.476283 (+ 448us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:44.477051 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:08:44.477067 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:08:44.480102 (+ 3035us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:08:44.482464 (+ 2362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:44.482470 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:44.482475 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:44.482714 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:44.483045 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:44.483051 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:44.483057 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:44.483123 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:44.483567 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:08:44.483573 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:08:44.483657 (+ 84us) client_negotiation.cc:769] Sending connection context
0903 08:08:44.483806 (+ 149us) client_negotiation.cc:240] Negotiation successful
0903 08:08:44.483945 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":983,"mutex_wait_us":27,"spinlock_wait_cycles":14464,"thread_start_us":124,"threads_started":1}
I20250903 08:08:44.484118 20732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:44.470817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.194:59201 (local address 127.12.201.252:35949)
0903 08:08:44.470963 (+ 146us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:44.470968 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:44.472307 (+ 1339us) server_negotiation.cc:408] Connection header received
0903 08:08:44.472870 (+ 563us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:44.472873 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:44.472926 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:44.473010 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:44.474432 (+ 1422us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.475225 (+ 793us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:44.475961 (+ 736us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.476152 (+ 191us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:44.480248 (+ 4096us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:44.480268 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:44.480272 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:08:44.480301 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:44.482331 (+ 2030us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:44.482860 (+ 529us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:44.482863 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:44.482865 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:44.482919 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:44.483219 (+ 300us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:44.483224 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:44.483228 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:44.483423 (+ 195us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:44.483517 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:44.483875 (+ 358us) server_negotiation.cc:300] Negotiation successful
0903 08:08:44.483986 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20250903 08:08:44.484354 21000 heartbeater.cc:344] Connected to a master server at 127.12.201.254:34143
I20250903 08:08:44.484588 21000 heartbeater.cc:461] Registering TS with master...
I20250903 08:08:44.484997 20999 heartbeater.cc:344] Connected to a master server at 127.12.201.252:35949
I20250903 08:08:44.485091 20999 heartbeater.cc:461] Registering TS with master...
I20250903 08:08:44.485129 21000 heartbeater.cc:507] Master 127.12.201.254:34143 requested a full tablet report, sending...
I20250903 08:08:44.485323 20999 heartbeater.cc:507] Master 127.12.201.252:35949 requested a full tablet report, sending...
I20250903 08:08:44.486158 20670 ts_manager.cc:194] Registered new tserver with Master: 2b6d679b81794eac94e52a9454286dcd (127.12.201.194:41585)
I20250903 08:08:44.486258 19873 ts_manager.cc:194] Re-registered known tserver with Master: 2b6d679b81794eac94e52a9454286dcd (127.12.201.194:41585)
I20250903 08:08:44.487043 19873 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.201.194@KRBTEST.COM'} at 127.12.201.194:51679
I20250903 08:08:44.487391 13095 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUML90r/build/debug/bin/kudu
/tmp/dist-test-taskUML90r/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.201.195:35143
--local_ip_for_outbound_sockets=127.12.201.195
--tserver_master_addrs=127.12.201.254:34143
--webserver_port=46171
--webserver_interface=127.12.201.195
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/krb5kdc/oryx.keytab
--principal=oryx/127.12.201.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.201.212:37903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.12.201.254:34143,127.12.201.253:42205
--tserver_master_addrs=127.12.201.254:34143,127.12.201.253:42205,127.12.201.252:35949 with env {KRB5CCNAME=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20250903 08:08:44.490923 21004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:44.471986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.253:42205 (local address 127.12.201.194:52759)
0903 08:08:44.472299 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:44.472314 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:44.472704 (+ 390us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:44.473174 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:44.473181 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:44.474198 (+ 1017us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:08:44.474460 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:44.474467 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.475209 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:44.475213 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:44.475807 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:44.475817 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.476283 (+ 466us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:44.476943 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:08:44.476955 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:08:44.487232 (+ 10277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:08:44.489415 (+ 2183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:44.489421 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:44.489424 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:44.489651 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:44.489899 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:44.489902 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:44.489904 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:44.489972 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:44.490448 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:08:44.490450 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:08:44.490541 (+ 91us) client_negotiation.cc:769] Sending connection context
0903 08:08:44.490626 (+ 85us) client_negotiation.cc:240] Negotiation successful
0903 08:08:44.490738 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"spinlock_wait_cycles":1280,"thread_start_us":77,"threads_started":1}
I20250903 08:08:44.491124 20729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:44.472155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.194:52759 (local address 127.12.201.253:42205)
0903 08:08:44.472304 (+ 149us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:44.472309 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:44.472382 (+ 73us) server_negotiation.cc:408] Connection header received
0903 08:08:44.472905 (+ 523us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:44.472909 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:44.472963 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:44.473016 (+ 53us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:44.474574 (+ 1558us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.475099 (+ 525us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:44.475962 (+ 863us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.476152 (+ 190us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:44.487411 (+ 11259us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:44.487433 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:44.487437 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:08:44.487469 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:44.489295 (+ 1826us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:44.489758 (+ 463us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:44.489767 (+ 9us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:44.489768 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:44.489814 (+ 46us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:44.490113 (+ 299us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:44.490119 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:44.490124 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:44.490333 (+ 209us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:44.490455 (+ 122us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:44.490714 (+ 259us) server_negotiation.cc:300] Negotiation successful
0903 08:08:44.490862 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20250903 08:08:44.491781 21001 heartbeater.cc:344] Connected to a master server at 127.12.201.253:42205
I20250903 08:08:44.491881 21001 heartbeater.cc:461] Registering TS with master...
I20250903 08:08:44.492210 21001 heartbeater.cc:507] Master 127.12.201.253:42205 requested a full tablet report, sending...
I20250903 08:08:44.493041 20486 ts_manager.cc:194] Registered new tserver with Master: 2b6d679b81794eac94e52a9454286dcd (127.12.201.194:41585)
W20250903 08:08:44.596884 21006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:08:44.597133 21006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:08:44.597225 21006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:08:44.600575 21006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 08:08:44.600652 21006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:08:44.600764 21006 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.201.195
I20250903 08:08:44.604954 21006 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.201.212:37903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/krb5kdc/oryx.keytab
--principal=oryx/127.12.201.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.201.195:35143
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.201.195
--webserver_port=46171
--webserver_require_spnego=true
--tserver_master_addrs=127.12.201.254:34143,127.12.201.253:42205,127.12.201.252:35949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21006
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.201.195
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 07:50:08 UTC on 5fd53c4cbb9d
build id 7874
I20250903 08:08:44.606069 21006 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:08:44.606921 21006 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:08:44.613539 21011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:08:44.613688 21006 server_base.cc:1047] running on GCE node
W20250903 08:08:44.613566 21012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:44.613538 21014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:08:44.614229 21006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:08:44.614763 21006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:08:44.615916 21006 hybrid_clock.cc:648] HybridClock initialized: now 1756886924615902 us; error 27 us; skew 500 ppm
Sep 03 08:08:44 dist-test-slave-27r9 krb5kdc[19827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886924, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.201.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 08:08:44.618531 21006 init.cc:377] Logged in from keytab as oryx/127.12.201.195@KRBTEST.COM (short username oryx)
I20250903 08:08:44.619613 21006 webserver.cc:480] Webserver started at http://127.12.201.195:46171/ using document root <none> and password file <none>
I20250903 08:08:44.620191 21006 fs_manager.cc:362] Metadata directory not provided
I20250903 08:08:44.620251 21006 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:08:44.623682 21006 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20250903 08:08:44.625608 21021 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:44.626595 21006 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250903 08:08:44.626698 21006 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-2/wal
uuid: "509ae480ec1c4aacb55daff15185c729"
format_stamp: "Formatted at 2025-09-03 08:08:42 on dist-test-slave-27r9"
server_key: "d2cf5d273e190c2a2afbae9fe5221c90"
server_key_iv: "f3c4c56b9ebb13f0f92df68e568dcea9"
server_key_version: "encryptionkey@0"
I20250903 08:08:44.627159 21006 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:08:44.646240 21006 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:08:44.649250 21006 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:08:44.649432 21006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:08:44.649976 21006 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 08:08:44.650838 21006 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 08:08:44.650888 21006 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:44.650947 21006 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 08:08:44.650974 21006 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:44.659520 21006 rpc_server.cc:307] RPC server started. Bound to: 127.12.201.195:35143
I20250903 08:08:44.659560 21134 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.201.195:35143 every 8 connection(s)
I20250903 08:08:44.660544 21006 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-2/data/info.pb
I20250903 08:08:44.663461 13095 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUML90r/build/debug/bin/kudu as pid 21006
Sep 03 08:08:44 dist-test-slave-27r9 krb5kdc[19827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886924, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.201.195@KRBTEST.COM for oryx/127.12.201.253@KRBTEST.COM
Sep 03 08:08:44 dist-test-slave-27r9 krb5kdc[19827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886924, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.201.195@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM
Sep 03 08:08:44 dist-test-slave-27r9 krb5kdc[19827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886924, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.201.195@KRBTEST.COM for oryx/127.12.201.252@KRBTEST.COM
I20250903 08:08:44.675635 20729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:44.662146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.195:46063 (local address 127.12.201.253:42205)
0903 08:08:44.662278 (+ 132us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:44.662283 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:44.663079 (+ 796us) server_negotiation.cc:408] Connection header received
0903 08:08:44.664144 (+ 1065us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:44.664148 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:44.664211 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:44.664290 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:44.666195 (+ 1905us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.666963 (+ 768us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:44.667909 (+ 946us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.668099 (+ 190us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:44.671391 (+ 3292us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:44.671410 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:44.671414 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:08:44.671442 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:44.673314 (+ 1872us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:44.673963 (+ 649us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:44.673969 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:44.673972 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:44.674024 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:44.674364 (+ 340us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:44.674370 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:44.674373 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:44.674566 (+ 193us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:44.674638 (+ 72us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:44.675345 (+ 707us) server_negotiation.cc:300] Negotiation successful
0903 08:08:44.675464 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20250903 08:08:44.676692 21139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:44.662462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:34143 (local address 127.12.201.195:44679)
0903 08:08:44.662923 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:44.662966 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:44.664259 (+ 1293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:44.664668 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:44.664671 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:44.665550 (+ 879us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:08:44.666030 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:44.666045 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.667159 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:44.667162 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:44.667784 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:44.667793 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.668270 (+ 477us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:44.669141 (+ 871us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:08:44.669163 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:08:44.671633 (+ 2470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:08:44.674285 (+ 2652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:44.674291 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:44.674296 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:44.674527 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:44.674782 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:44.674785 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:44.674802 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:44.674850 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:44.675264 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:08:44.675268 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:08:44.675344 (+ 76us) client_negotiation.cc:769] Sending connection context
0903 08:08:44.675462 (+ 118us) client_negotiation.cc:240] Negotiation successful
0903 08:08:44.675751 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"spinlock_wait_cycles":24960,"thread_start_us":113,"threads_started":1}
I20250903 08:08:44.676705 21140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:44.662462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.253:42205 (local address 127.12.201.195:46063)
0903 08:08:44.662923 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:44.662966 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:44.663930 (+ 964us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:44.664576 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:44.664586 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:44.665241 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:08:44.666028 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:44.666044 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.667135 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:44.667138 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:44.667772 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:44.667781 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.668270 (+ 489us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:44.668937 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:08:44.668973 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:08:44.671245 (+ 2272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:08:44.673472 (+ 2227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:44.673480 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:44.673497 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:44.673837 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:44.674136 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:44.674142 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:44.674145 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:44.674261 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:44.674717 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:08:44.674726 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:08:44.675054 (+ 328us) client_negotiation.cc:769] Sending connection context
0903 08:08:44.675870 (+ 816us) client_negotiation.cc:240] Negotiation successful
0903 08:08:44.675974 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"mutex_wait_us":43,"spinlock_wait_cycles":31360,"thread_start_us":119,"threads_started":1}
I20250903 08:08:44.678184 21137 heartbeater.cc:344] Connected to a master server at 127.12.201.253:42205
I20250903 08:08:44.678457 21137 heartbeater.cc:461] Registering TS with master...
I20250903 08:08:44.678541 20731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:44.662201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.195:44679 (local address 127.12.201.254:34143)
0903 08:08:44.662345 (+ 144us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:44.662350 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:44.663085 (+ 735us) server_negotiation.cc:408] Connection header received
0903 08:08:44.664371 (+ 1286us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:44.664377 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:44.664445 (+ 68us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:44.664529 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:44.666196 (+ 1667us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.667009 (+ 813us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:44.667919 (+ 910us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.668088 (+ 169us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:44.671747 (+ 3659us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:44.671765 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:44.671772 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:08:44.671811 (+ 39us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:44.674180 (+ 2369us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:44.674638 (+ 458us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:44.674642 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:44.674645 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:44.674694 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:44.674944 (+ 250us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:44.674947 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:44.674948 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:44.675126 (+ 178us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:44.675218 (+ 92us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:44.678149 (+ 2931us) server_negotiation.cc:300] Negotiation successful
0903 08:08:44.678288 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20250903 08:08:44.679165 21137 heartbeater.cc:507] Master 127.12.201.253:42205 requested a full tablet report, sending...
I20250903 08:08:44.679191 21136 heartbeater.cc:344] Connected to a master server at 127.12.201.254:34143
I20250903 08:08:44.679270 21136 heartbeater.cc:461] Registering TS with master...
I20250903 08:08:44.679466 21136 heartbeater.cc:507] Master 127.12.201.254:34143 requested a full tablet report, sending...
I20250903 08:08:44.680289 19873 ts_manager.cc:194] Re-registered known tserver with Master: 509ae480ec1c4aacb55daff15185c729 (127.12.201.195:35143)
I20250903 08:08:44.680392 20486 ts_manager.cc:194] Registered new tserver with Master: 509ae480ec1c4aacb55daff15185c729 (127.12.201.195:35143)
I20250903 08:08:44.680977 19873 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.201.195@KRBTEST.COM'} at 127.12.201.195:44679
I20250903 08:08:44.681139 21148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:44.669467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38758 (local address 127.12.201.253:42205)
0903 08:08:44.672478 (+ 3011us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:44.672483 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:44.672496 (+ 13us) server_negotiation.cc:408] Connection header received
0903 08:08:44.672533 (+ 37us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:44.672536 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:44.672594 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:44.672690 (+ 96us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:44.674070 (+ 1380us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.674880 (+ 810us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:44.676110 (+ 1230us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.676305 (+ 195us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:44.677314 (+ 1009us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:44.677334 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:44.677339 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:08:44.677368 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:44.679347 (+ 1979us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:44.679801 (+ 454us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:44.679804 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:44.679806 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:44.679851 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:44.680108 (+ 257us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:44.680111 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:44.680113 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:44.680273 (+ 160us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:44.680396 (+ 123us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:44.680891 (+ 495us) server_negotiation.cc:300] Negotiation successful
0903 08:08:44.681003 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2927,"thread_start_us":91,"threads_started":1}
I20250903 08:08:44.682754 20732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:44.663784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.195:56331 (local address 127.12.201.252:35949)
0903 08:08:44.663939 (+ 155us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:44.663944 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:44.664538 (+ 594us) server_negotiation.cc:408] Connection header received
0903 08:08:44.664694 (+ 156us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:44.664700 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:44.664759 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:44.664824 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:44.666194 (+ 1370us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.666963 (+ 769us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:44.667908 (+ 945us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.668099 (+ 191us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:44.672160 (+ 4061us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:44.672184 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:44.672187 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:08:44.672217 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:44.674213 (+ 1996us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:44.681524 (+ 7311us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:44.681530 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:44.681534 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:44.681583 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:44.681865 (+ 282us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:44.681868 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:44.681870 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:44.682033 (+ 163us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:44.682107 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:44.682449 (+ 342us) server_negotiation.cc:300] Negotiation successful
0903 08:08:44.682587 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20250903 08:08:44.683374 21141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:44.663436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.252:35949 (local address 127.12.201.195:56331)
0903 08:08:44.664468 (+ 1032us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:44.664481 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:44.664593 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:44.664962 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:44.664968 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:44.665241 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:08:44.666028 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:44.666044 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.667135 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:44.667139 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:44.667774 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:44.667781 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.668270 (+ 489us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:44.669474 (+ 1204us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:08:44.669492 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:08:44.672042 (+ 2550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:08:44.674324 (+ 2282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:44.674328 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:44.674330 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:44.681436 (+ 7106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:44.681688 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:44.681691 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:44.681693 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:44.681748 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:44.682183 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:08:44.682186 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:08:44.682269 (+ 83us) client_negotiation.cc:769] Sending connection context
0903 08:08:44.682968 (+ 699us) client_negotiation.cc:240] Negotiation successful
0903 08:08:44.683163 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":973,"spinlock_wait_cycles":8448,"thread_start_us":79,"threads_started":1}
I20250903 08:08:44.683509 21150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:44.670480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56930 (local address 127.12.201.252:35949)
0903 08:08:44.670729 (+ 249us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:44.670734 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:44.673759 (+ 3025us) server_negotiation.cc:408] Connection header received
0903 08:08:44.673808 (+ 49us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:44.673811 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:44.673864 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:44.673949 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:44.675401 (+ 1452us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.676166 (+ 765us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:44.677230 (+ 1064us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.677483 (+ 253us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:44.678618 (+ 1135us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:44.678637 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:44.678640 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:08:44.678668 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:44.680719 (+ 2051us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:44.681165 (+ 446us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:44.681168 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:44.681170 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:44.681220 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:44.682323 (+ 1103us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:44.682330 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:44.682335 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:44.682492 (+ 157us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:44.682974 (+ 482us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:44.683230 (+ 256us) server_negotiation.cc:300] Negotiation successful
0903 08:08:44.683341 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":60,"threads_started":1}
I20250903 08:08:44.683907 21135 heartbeater.cc:344] Connected to a master server at 127.12.201.252:35949
I20250903 08:08:44.683992 21135 heartbeater.cc:461] Registering TS with master...
I20250903 08:08:44.684166 21135 heartbeater.cc:507] Master 127.12.201.252:35949 requested a full tablet report, sending...
I20250903 08:08:44.684641 21146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:44.665385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37176 (local address 127.12.201.254:34143)
0903 08:08:44.665982 (+ 597us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:44.665986 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:44.666330 (+ 344us) server_negotiation.cc:408] Connection header received
0903 08:08:44.666365 (+ 35us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:44.666368 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:44.666411 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:44.666478 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:44.667559 (+ 1081us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.668263 (+ 704us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:44.669027 (+ 764us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.669754 (+ 727us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:44.670581 (+ 827us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:44.670605 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:44.670608 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:08:44.670634 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:44.682522 (+ 11888us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:44.682933 (+ 411us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:44.682937 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:44.682940 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:44.683010 (+ 70us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:44.683399 (+ 389us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:44.683403 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:44.683405 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:44.683929 (+ 524us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:44.684060 (+ 131us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:44.684341 (+ 281us) server_negotiation.cc:300] Negotiation successful
0903 08:08:44.684465 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":72,"threads_started":1}
I20250903 08:08:44.684860 20669 ts_manager.cc:194] Registered new tserver with Master: 509ae480ec1c4aacb55daff15185c729 (127.12.201.195:35143)
I20250903 08:08:44.687175 19873 catalog_manager.cc:2232] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37176:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 08:08:44.689381 19873 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 08:08:44.705135 21164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:44.701369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.193:43289 (local address 127.0.0.1:45866)
0903 08:08:44.701916 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:44.701931 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:44.702010 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:44.702788 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:44.702791 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:44.702811 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 08:08:44.703033 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:44.703039 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.704065 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:44.704068 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:44.704724 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:44.704733 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.704856 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:44.704879 (+ 23us) client_negotiation.cc:769] Sending connection context
0903 08:08:44.704932 (+ 53us) client_negotiation.cc:240] Negotiation successful
0903 08:08:44.704992 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":105,"threads_started":1}
I20250903 08:08:44.705340 21162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:44.700837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.194:41585 (local address 127.0.0.1:49014)
0903 08:08:44.701622 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:44.701641 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:44.702055 (+ 414us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:44.702489 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:44.702492 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:44.702510 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 08:08:44.702764 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:44.702774 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.703984 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:44.703988 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:44.704940 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:44.704949 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.705056 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:44.705075 (+ 19us) client_negotiation.cc:769] Sending connection context
0903 08:08:44.705130 (+ 55us) client_negotiation.cc:240] Negotiation successful
0903 08:08:44.705192 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":712,"thread_start_us":110,"threads_started":1}
I20250903 08:08:44.705755 21163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:44.701572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45866 (local address 127.12.201.193:43289)
0903 08:08:44.702397 (+ 825us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:44.702403 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:44.702421 (+ 18us) server_negotiation.cc:408] Connection header received
0903 08:08:44.702513 (+ 92us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:44.702520 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:44.702662 (+ 142us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:44.702791 (+ 129us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 08:08:44.703141 (+ 350us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.703941 (+ 800us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:44.704866 (+ 925us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.705398 (+ 532us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:44.705522 (+ 124us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:44.705582 (+ 60us) server_negotiation.cc:300] Negotiation successful
0903 08:08:44.705650 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":732,"thread_start_us":102,"threads_started":1}
I20250903 08:08:44.706593 21160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:44.701695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.195:35143 (local address 127.0.0.1:36654)
0903 08:08:44.702044 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:44.702058 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:44.702164 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:44.702943 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:44.702946 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:44.702965 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 08:08:44.703231 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:44.703238 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.705552 (+ 2314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:44.705555 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:44.706243 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:44.706253 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.706365 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:44.706380 (+ 15us) client_negotiation.cc:769] Sending connection context
0903 08:08:44.706428 (+ 48us) client_negotiation.cc:240] Negotiation successful
0903 08:08:44.706496 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":80,"threads_started":1}
I20250903 08:08:44.707305 21165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:44.701822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36654 (local address 127.12.201.195:35143)
0903 08:08:44.702166 (+ 344us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:44.702174 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:44.702198 (+ 24us) server_negotiation.cc:408] Connection header received
0903 08:08:44.702284 (+ 86us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:44.702289 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:44.702459 (+ 170us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:44.702590 (+ 131us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 08:08:44.704002 (+ 1412us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.705068 (+ 1066us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:44.706361 (+ 1293us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.706915 (+ 554us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:44.707036 (+ 121us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:44.707098 (+ 62us) server_negotiation.cc:300] Negotiation successful
0903 08:08:44.707180 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":106,"threads_started":1}
I20250903 08:08:44.707638 21161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:44.700988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49014 (local address 127.12.201.194:41585)
0903 08:08:44.701871 (+ 883us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:44.701878 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:44.701904 (+ 26us) server_negotiation.cc:408] Connection header received
0903 08:08:44.702212 (+ 308us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:44.702217 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:44.702369 (+ 152us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:44.702498 (+ 129us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 08:08:44.702895 (+ 397us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.703872 (+ 977us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:44.706762 (+ 2890us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.707282 (+ 520us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:44.707365 (+ 83us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:44.707450 (+ 85us) server_negotiation.cc:300] Negotiation successful
0903 08:08:44.707533 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":87,"threads_started":1}
I20250903 08:08:44.708518 20797 tablet_service.cc:1468] Processing CreateTablet for tablet 09456e56b7ab4ce498c3e90b9daa7425 (DEFAULT_TABLE table=test-table [id=c2946fcd2cf74576aba0f351f49d339b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:08:44.709527 20797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09456e56b7ab4ce498c3e90b9daa7425. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:08:44.709407 21069 tablet_service.cc:1468] Processing CreateTablet for tablet 09456e56b7ab4ce498c3e90b9daa7425 (DEFAULT_TABLE table=test-table [id=c2946fcd2cf74576aba0f351f49d339b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:08:44.710165 21069 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09456e56b7ab4ce498c3e90b9daa7425. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:08:44.709903 20933 tablet_service.cc:1468] Processing CreateTablet for tablet 09456e56b7ab4ce498c3e90b9daa7425 (DEFAULT_TABLE table=test-table [id=c2946fcd2cf74576aba0f351f49d339b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:08:44.710666 20933 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09456e56b7ab4ce498c3e90b9daa7425. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:08:44.715989 21166 tablet_bootstrap.cc:492] T 09456e56b7ab4ce498c3e90b9daa7425 P 509ae480ec1c4aacb55daff15185c729: Bootstrap starting.
I20250903 08:08:44.716447 21167 tablet_bootstrap.cc:492] T 09456e56b7ab4ce498c3e90b9daa7425 P a16339bb0a5e4d3e93e3083c087b5fe7: Bootstrap starting.
I20250903 08:08:44.716965 21168 tablet_bootstrap.cc:492] T 09456e56b7ab4ce498c3e90b9daa7425 P 2b6d679b81794eac94e52a9454286dcd: Bootstrap starting.
I20250903 08:08:44.719075 21168 tablet_bootstrap.cc:654] T 09456e56b7ab4ce498c3e90b9daa7425 P 2b6d679b81794eac94e52a9454286dcd: Neither blocks nor log segments found. Creating new log.
I20250903 08:08:44.719195 21167 tablet_bootstrap.cc:654] T 09456e56b7ab4ce498c3e90b9daa7425 P a16339bb0a5e4d3e93e3083c087b5fe7: Neither blocks nor log segments found. Creating new log.
I20250903 08:08:44.719195 21166 tablet_bootstrap.cc:654] T 09456e56b7ab4ce498c3e90b9daa7425 P 509ae480ec1c4aacb55daff15185c729: Neither blocks nor log segments found. Creating new log.
I20250903 08:08:44.719863 21168 log.cc:826] T 09456e56b7ab4ce498c3e90b9daa7425 P 2b6d679b81794eac94e52a9454286dcd: Log is configured to *not* fsync() on all Append() calls
I20250903 08:08:44.720005 21167 log.cc:826] T 09456e56b7ab4ce498c3e90b9daa7425 P a16339bb0a5e4d3e93e3083c087b5fe7: Log is configured to *not* fsync() on all Append() calls
I20250903 08:08:44.720005 21166 log.cc:826] T 09456e56b7ab4ce498c3e90b9daa7425 P 509ae480ec1c4aacb55daff15185c729: Log is configured to *not* fsync() on all Append() calls
I20250903 08:08:44.721556 21166 tablet_bootstrap.cc:492] T 09456e56b7ab4ce498c3e90b9daa7425 P 509ae480ec1c4aacb55daff15185c729: No bootstrap required, opened a new log
I20250903 08:08:44.721562 21167 tablet_bootstrap.cc:492] T 09456e56b7ab4ce498c3e90b9daa7425 P a16339bb0a5e4d3e93e3083c087b5fe7: No bootstrap required, opened a new log
I20250903 08:08:44.721721 21166 ts_tablet_manager.cc:1397] T 09456e56b7ab4ce498c3e90b9daa7425 P 509ae480ec1c4aacb55daff15185c729: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20250903 08:08:44.721726 21167 ts_tablet_manager.cc:1397] T 09456e56b7ab4ce498c3e90b9daa7425 P a16339bb0a5e4d3e93e3083c087b5fe7: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20250903 08:08:44.722699 21168 tablet_bootstrap.cc:492] T 09456e56b7ab4ce498c3e90b9daa7425 P 2b6d679b81794eac94e52a9454286dcd: No bootstrap required, opened a new log
I20250903 08:08:44.722846 21168 ts_tablet_manager.cc:1397] T 09456e56b7ab4ce498c3e90b9daa7425 P 2b6d679b81794eac94e52a9454286dcd: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20250903 08:08:44.724012 21167 raft_consensus.cc:357] T 09456e56b7ab4ce498c3e90b9daa7425 P a16339bb0a5e4d3e93e3083c087b5fe7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b6d679b81794eac94e52a9454286dcd" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 41585 } } peers { permanent_uuid: "a16339bb0a5e4d3e93e3083c087b5fe7" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 43289 } } peers { permanent_uuid: "509ae480ec1c4aacb55daff15185c729" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 35143 } }
I20250903 08:08:44.724212 21167 raft_consensus.cc:383] T 09456e56b7ab4ce498c3e90b9daa7425 P a16339bb0a5e4d3e93e3083c087b5fe7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:08:44.724278 21167 raft_consensus.cc:738] T 09456e56b7ab4ce498c3e90b9daa7425 P a16339bb0a5e4d3e93e3083c087b5fe7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a16339bb0a5e4d3e93e3083c087b5fe7, State: Initialized, Role: FOLLOWER
I20250903 08:08:44.724606 21166 raft_consensus.cc:357] T 09456e56b7ab4ce498c3e90b9daa7425 P 509ae480ec1c4aacb55daff15185c729 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b6d679b81794eac94e52a9454286dcd" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 41585 } } peers { permanent_uuid: "a16339bb0a5e4d3e93e3083c087b5fe7" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 43289 } } peers { permanent_uuid: "509ae480ec1c4aacb55daff15185c729" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 35143 } }
I20250903 08:08:44.724735 21167 consensus_queue.cc:260] T 09456e56b7ab4ce498c3e90b9daa7425 P a16339bb0a5e4d3e93e3083c087b5fe7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b6d679b81794eac94e52a9454286dcd" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 41585 } } peers { permanent_uuid: "a16339bb0a5e4d3e93e3083c087b5fe7" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 43289 } } peers { permanent_uuid: "509ae480ec1c4aacb55daff15185c729" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 35143 } }
I20250903 08:08:44.724836 21166 raft_consensus.cc:383] T 09456e56b7ab4ce498c3e90b9daa7425 P 509ae480ec1c4aacb55daff15185c729 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:08:44.724889 21166 raft_consensus.cc:738] T 09456e56b7ab4ce498c3e90b9daa7425 P 509ae480ec1c4aacb55daff15185c729 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 509ae480ec1c4aacb55daff15185c729, State: Initialized, Role: FOLLOWER
I20250903 08:08:44.725555 21167 ts_tablet_manager.cc:1428] T 09456e56b7ab4ce498c3e90b9daa7425 P a16339bb0a5e4d3e93e3083c087b5fe7: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20250903 08:08:44.725734 20864 heartbeater.cc:499] Master 127.12.201.254:34143 was elected leader, sending a full tablet report...
I20250903 08:08:44.725656 21168 raft_consensus.cc:357] T 09456e56b7ab4ce498c3e90b9daa7425 P 2b6d679b81794eac94e52a9454286dcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b6d679b81794eac94e52a9454286dcd" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 41585 } } peers { permanent_uuid: "a16339bb0a5e4d3e93e3083c087b5fe7" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 43289 } } peers { permanent_uuid: "509ae480ec1c4aacb55daff15185c729" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 35143 } }
I20250903 08:08:44.725831 21168 raft_consensus.cc:383] T 09456e56b7ab4ce498c3e90b9daa7425 P 2b6d679b81794eac94e52a9454286dcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:08:44.725879 21168 raft_consensus.cc:738] T 09456e56b7ab4ce498c3e90b9daa7425 P 2b6d679b81794eac94e52a9454286dcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b6d679b81794eac94e52a9454286dcd, State: Initialized, Role: FOLLOWER
I20250903 08:08:44.725306 21166 consensus_queue.cc:260] T 09456e56b7ab4ce498c3e90b9daa7425 P 509ae480ec1c4aacb55daff15185c729 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b6d679b81794eac94e52a9454286dcd" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 41585 } } peers { permanent_uuid: "a16339bb0a5e4d3e93e3083c087b5fe7" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 43289 } } peers { permanent_uuid: "509ae480ec1c4aacb55daff15185c729" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 35143 } }
I20250903 08:08:44.726236 21168 consensus_queue.cc:260] T 09456e56b7ab4ce498c3e90b9daa7425 P 2b6d679b81794eac94e52a9454286dcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b6d679b81794eac94e52a9454286dcd" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 41585 } } peers { permanent_uuid: "a16339bb0a5e4d3e93e3083c087b5fe7" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 43289 } } peers { permanent_uuid: "509ae480ec1c4aacb55daff15185c729" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 35143 } }
I20250903 08:08:44.726897 21166 ts_tablet_manager.cc:1428] T 09456e56b7ab4ce498c3e90b9daa7425 P 509ae480ec1c4aacb55daff15185c729: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20250903 08:08:44.726897 21168 ts_tablet_manager.cc:1428] T 09456e56b7ab4ce498c3e90b9daa7425 P 2b6d679b81794eac94e52a9454286dcd: Time spent starting tablet: real 0.004s user 0.001s sys 0.004s
I20250903 08:08:44.727377 21000 heartbeater.cc:499] Master 127.12.201.254:34143 was elected leader, sending a full tablet report...
I20250903 08:08:44.728065 21136 heartbeater.cc:499] Master 127.12.201.254:34143 was elected leader, sending a full tablet report...
W20250903 08:08:44.782861 20866 tablet.cc:2378] T 09456e56b7ab4ce498c3e90b9daa7425 P a16339bb0a5e4d3e93e3083c087b5fe7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250903 08:08:44.805816 21172 raft_consensus.cc:491] T 09456e56b7ab4ce498c3e90b9daa7425 P a16339bb0a5e4d3e93e3083c087b5fe7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 08:08:44.806039 21172 raft_consensus.cc:513] T 09456e56b7ab4ce498c3e90b9daa7425 P a16339bb0a5e4d3e93e3083c087b5fe7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b6d679b81794eac94e52a9454286dcd" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 41585 } } peers { permanent_uuid: "a16339bb0a5e4d3e93e3083c087b5fe7" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 43289 } } peers { permanent_uuid: "509ae480ec1c4aacb55daff15185c729" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 35143 } }
I20250903 08:08:44.807190 21172 leader_election.cc:290] T 09456e56b7ab4ce498c3e90b9daa7425 P a16339bb0a5e4d3e93e3083c087b5fe7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2b6d679b81794eac94e52a9454286dcd (127.12.201.194:41585), 509ae480ec1c4aacb55daff15185c729 (127.12.201.195:35143)
I20250903 08:08:44.810686 21175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:44.807423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.195:35143 (local address 127.12.201.193:43287)
0903 08:08:44.807784 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:44.807800 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:44.807925 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:44.808216 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:44.808219 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:44.808240 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 08:08:44.808497 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:44.808506 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.809409 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:44.809412 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:44.810253 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:44.810262 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.810368 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:44.810391 (+ 23us) client_negotiation.cc:769] Sending connection context
0903 08:08:44.810440 (+ 49us) client_negotiation.cc:240] Negotiation successful
0903 08:08:44.810511 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":97,"threads_started":1}
I20250903 08:08:44.810817 21176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:44.807423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.194:41585 (local address 127.12.201.193:50983)
0903 08:08:44.807834 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:44.807847 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:44.807927 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:44.808460 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:44.808464 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:44.808481 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 08:08:44.808723 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:44.808730 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.809659 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:44.809662 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:44.810436 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:44.810444 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.810546 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:44.810559 (+ 13us) client_negotiation.cc:769] Sending connection context
0903 08:08:44.810604 (+ 45us) client_negotiation.cc:240] Negotiation successful
0903 08:08:44.810645 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"mutex_wait_us":20,"spinlock_wait_cycles":6400,"thread_start_us":79,"threads_started":1}
I20250903 08:08:44.811311 21161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:44.807523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.193:50983 (local address 127.12.201.194:41585)
0903 08:08:44.807680 (+ 157us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:44.807685 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:44.807909 (+ 224us) server_negotiation.cc:408] Connection header received
0903 08:08:44.808036 (+ 127us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:44.808040 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:44.808100 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:44.808206 (+ 106us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 08:08:44.808851 (+ 645us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.809567 (+ 716us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:44.810540 (+ 973us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.811044 (+ 504us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:44.811085 (+ 41us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:44.811138 (+ 53us) server_negotiation.cc:300] Negotiation successful
0903 08:08:44.811194 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20250903 08:08:44.811444 21165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:44.807523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.193:43287 (local address 127.12.201.195:35143)
0903 08:08:44.807680 (+ 157us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:44.807685 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:44.807884 (+ 199us) server_negotiation.cc:408] Connection header received
0903 08:08:44.808046 (+ 162us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:44.808049 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:44.808100 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:44.808198 (+ 98us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 08:08:44.808628 (+ 430us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.809311 (+ 683us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:44.810375 (+ 1064us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.811194 (+ 819us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:44.811227 (+ 33us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:44.811274 (+ 47us) server_negotiation.cc:300] Negotiation successful
0903 08:08:44.811327 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20250903 08:08:44.811888 20953 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "09456e56b7ab4ce498c3e90b9daa7425" candidate_uuid: "a16339bb0a5e4d3e93e3083c087b5fe7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b6d679b81794eac94e52a9454286dcd" is_pre_election: true
I20250903 08:08:44.811957 21089 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "09456e56b7ab4ce498c3e90b9daa7425" candidate_uuid: "a16339bb0a5e4d3e93e3083c087b5fe7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "509ae480ec1c4aacb55daff15185c729" is_pre_election: true
I20250903 08:08:44.812184 20953 raft_consensus.cc:2466] T 09456e56b7ab4ce498c3e90b9daa7425 P 2b6d679b81794eac94e52a9454286dcd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a16339bb0a5e4d3e93e3083c087b5fe7 in term 0.
I20250903 08:08:44.812214 21089 raft_consensus.cc:2466] T 09456e56b7ab4ce498c3e90b9daa7425 P 509ae480ec1c4aacb55daff15185c729 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a16339bb0a5e4d3e93e3083c087b5fe7 in term 0.
I20250903 08:08:44.812670 20750 leader_election.cc:304] T 09456e56b7ab4ce498c3e90b9daa7425 P a16339bb0a5e4d3e93e3083c087b5fe7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2b6d679b81794eac94e52a9454286dcd, 509ae480ec1c4aacb55daff15185c729, a16339bb0a5e4d3e93e3083c087b5fe7; no voters:
I20250903 08:08:44.812928 21172 raft_consensus.cc:2802] T 09456e56b7ab4ce498c3e90b9daa7425 P a16339bb0a5e4d3e93e3083c087b5fe7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 08:08:44.812999 21172 raft_consensus.cc:491] T 09456e56b7ab4ce498c3e90b9daa7425 P a16339bb0a5e4d3e93e3083c087b5fe7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 08:08:44.813030 21172 raft_consensus.cc:3058] T 09456e56b7ab4ce498c3e90b9daa7425 P a16339bb0a5e4d3e93e3083c087b5fe7 [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:08:44.813990 21172 raft_consensus.cc:513] T 09456e56b7ab4ce498c3e90b9daa7425 P a16339bb0a5e4d3e93e3083c087b5fe7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b6d679b81794eac94e52a9454286dcd" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 41585 } } peers { permanent_uuid: "a16339bb0a5e4d3e93e3083c087b5fe7" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 43289 } } peers { permanent_uuid: "509ae480ec1c4aacb55daff15185c729" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 35143 } }
I20250903 08:08:44.814338 21172 leader_election.cc:290] T 09456e56b7ab4ce498c3e90b9daa7425 P a16339bb0a5e4d3e93e3083c087b5fe7 [CANDIDATE]: Term 1 election: Requested vote from peers 2b6d679b81794eac94e52a9454286dcd (127.12.201.194:41585), 509ae480ec1c4aacb55daff15185c729 (127.12.201.195:35143)
I20250903 08:08:44.814714 20953 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "09456e56b7ab4ce498c3e90b9daa7425" candidate_uuid: "a16339bb0a5e4d3e93e3083c087b5fe7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b6d679b81794eac94e52a9454286dcd"
I20250903 08:08:44.814731 21089 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "09456e56b7ab4ce498c3e90b9daa7425" candidate_uuid: "a16339bb0a5e4d3e93e3083c087b5fe7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "509ae480ec1c4aacb55daff15185c729"
I20250903 08:08:44.814829 21089 raft_consensus.cc:3058] T 09456e56b7ab4ce498c3e90b9daa7425 P 509ae480ec1c4aacb55daff15185c729 [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:08:44.814838 20953 raft_consensus.cc:3058] T 09456e56b7ab4ce498c3e90b9daa7425 P 2b6d679b81794eac94e52a9454286dcd [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:08:44.815846 21089 raft_consensus.cc:2466] T 09456e56b7ab4ce498c3e90b9daa7425 P 509ae480ec1c4aacb55daff15185c729 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a16339bb0a5e4d3e93e3083c087b5fe7 in term 1.
I20250903 08:08:44.816154 20751 leader_election.cc:304] T 09456e56b7ab4ce498c3e90b9daa7425 P a16339bb0a5e4d3e93e3083c087b5fe7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 509ae480ec1c4aacb55daff15185c729, a16339bb0a5e4d3e93e3083c087b5fe7; no voters:
I20250903 08:08:44.816361 21172 raft_consensus.cc:2802] T 09456e56b7ab4ce498c3e90b9daa7425 P a16339bb0a5e4d3e93e3083c087b5fe7 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 08:08:44.816385 20953 raft_consensus.cc:2466] T 09456e56b7ab4ce498c3e90b9daa7425 P 2b6d679b81794eac94e52a9454286dcd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a16339bb0a5e4d3e93e3083c087b5fe7 in term 1.
I20250903 08:08:44.816551 21172 raft_consensus.cc:695] T 09456e56b7ab4ce498c3e90b9daa7425 P a16339bb0a5e4d3e93e3083c087b5fe7 [term 1 LEADER]: Becoming Leader. State: Replica: a16339bb0a5e4d3e93e3083c087b5fe7, State: Running, Role: LEADER
I20250903 08:08:44.817234 21172 consensus_queue.cc:237] T 09456e56b7ab4ce498c3e90b9daa7425 P a16339bb0a5e4d3e93e3083c087b5fe7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b6d679b81794eac94e52a9454286dcd" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 41585 } } peers { permanent_uuid: "a16339bb0a5e4d3e93e3083c087b5fe7" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 43289 } } peers { permanent_uuid: "509ae480ec1c4aacb55daff15185c729" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 35143 } }
I20250903 08:08:44.820696 19873 catalog_manager.cc:5582] T 09456e56b7ab4ce498c3e90b9daa7425 P a16339bb0a5e4d3e93e3083c087b5fe7 reported cstate change: term changed from 0 to 1, leader changed from <none> to a16339bb0a5e4d3e93e3083c087b5fe7 (127.12.201.193). New cstate: current_term: 1 leader_uuid: "a16339bb0a5e4d3e93e3083c087b5fe7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b6d679b81794eac94e52a9454286dcd" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 41585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a16339bb0a5e4d3e93e3083c087b5fe7" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 43289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "509ae480ec1c4aacb55daff15185c729" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 35143 } health_report { overall_health: UNKNOWN } } }
I20250903 08:08:44.860841 21163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:44.857843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45870 (local address 127.12.201.193:43289)
0903 08:08:44.858006 (+ 163us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:44.858011 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:44.858027 (+ 16us) server_negotiation.cc:408] Connection header received
0903 08:08:44.858102 (+ 75us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:44.858105 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:44.858157 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:44.858245 (+ 88us) server_negotiation.cc:227] Negotiated authn=TOKEN
0903 08:08:44.858593 (+ 348us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.859117 (+ 524us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:44.859761 (+ 644us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.859940 (+ 179us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:44.860031 (+ 91us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0903 08:08:44.860462 (+ 431us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0903 08:08:44.860586 (+ 124us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:44.860644 (+ 58us) server_negotiation.cc:300] Negotiation successful
0903 08:08:44.860694 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20250903 08:08:44.866199 21089 raft_consensus.cc:1273] T 09456e56b7ab4ce498c3e90b9daa7425 P 509ae480ec1c4aacb55daff15185c729 [term 1 FOLLOWER]: Refusing update from remote peer a16339bb0a5e4d3e93e3083c087b5fe7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 08:08:44.866197 20953 raft_consensus.cc:1273] T 09456e56b7ab4ce498c3e90b9daa7425 P 2b6d679b81794eac94e52a9454286dcd [term 1 FOLLOWER]: Refusing update from remote peer a16339bb0a5e4d3e93e3083c087b5fe7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 08:08:44.866845 21172 consensus_queue.cc:1035] T 09456e56b7ab4ce498c3e90b9daa7425 P a16339bb0a5e4d3e93e3083c087b5fe7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b6d679b81794eac94e52a9454286dcd" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 41585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 08:08:44.867064 21177 consensus_queue.cc:1035] T 09456e56b7ab4ce498c3e90b9daa7425 P a16339bb0a5e4d3e93e3083c087b5fe7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "509ae480ec1c4aacb55daff15185c729" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 35143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 08:08:44.876405 21181 mvcc.cc:204] Tried to move back new op lower bound from 7196208844245065728 to 7196208844053815296. Current Snapshot: MvccSnapshot[applied={T|T < 7196208844245065728}]
I20250903 08:08:44.881170 21182 mvcc.cc:204] Tried to move back new op lower bound from 7196208844245065728 to 7196208844053815296. Current Snapshot: MvccSnapshot[applied={T|T < 7196208844245065728}]
I20250903 08:08:44.886014 19873 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37176:
table { table_name: "test-table" } modify_external_catalogs: true
I20250903 08:08:44.886162 19873 catalog_manager.cc:2730] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37176:
table { table_name: "test-table" } modify_external_catalogs: true
I20250903 08:08:44.890071 19873 catalog_manager.cc:5869] T 00000000000000000000000000000000 P ff08db2a004746a6ab7e1d0fa91eed38: Sending DeleteTablet for 3 replicas of tablet 09456e56b7ab4ce498c3e90b9daa7425
I20250903 08:08:44.891239 21069 tablet_service.cc:1515] Processing DeleteTablet for tablet 09456e56b7ab4ce498c3e90b9daa7425 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 08:08:44 UTC) from {username='oryx', principal='oryx/127.12.201.254@KRBTEST.COM'} at 127.0.0.1:36654
I20250903 08:08:44.891237 20797 tablet_service.cc:1515] Processing DeleteTablet for tablet 09456e56b7ab4ce498c3e90b9daa7425 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 08:08:44 UTC) from {username='oryx', principal='oryx/127.12.201.254@KRBTEST.COM'} at 127.0.0.1:45866
I20250903 08:08:44.891731 21189 tablet_replica.cc:331] T 09456e56b7ab4ce498c3e90b9daa7425 P 509ae480ec1c4aacb55daff15185c729: stopping tablet replica
I20250903 08:08:44.891731 21190 tablet_replica.cc:331] T 09456e56b7ab4ce498c3e90b9daa7425 P a16339bb0a5e4d3e93e3083c087b5fe7: stopping tablet replica
I20250903 08:08:44.892015 21190 raft_consensus.cc:2241] T 09456e56b7ab4ce498c3e90b9daa7425 P a16339bb0a5e4d3e93e3083c087b5fe7 [term 1 LEADER]: Raft consensus shutting down.
I20250903 08:08:44.892015 21189 raft_consensus.cc:2241] T 09456e56b7ab4ce498c3e90b9daa7425 P 509ae480ec1c4aacb55daff15185c729 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 08:08:44.892201 20933 tablet_service.cc:1515] Processing DeleteTablet for tablet 09456e56b7ab4ce498c3e90b9daa7425 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 08:08:44 UTC) from {username='oryx', principal='oryx/127.12.201.254@KRBTEST.COM'} at 127.0.0.1:49014
I20250903 08:08:44.892345 21190 raft_consensus.cc:2270] T 09456e56b7ab4ce498c3e90b9daa7425 P a16339bb0a5e4d3e93e3083c087b5fe7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 08:08:44.892345 21189 raft_consensus.cc:2270] T 09456e56b7ab4ce498c3e90b9daa7425 P 509ae480ec1c4aacb55daff15185c729 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 08:08:44.893901 21190 ts_tablet_manager.cc:1905] T 09456e56b7ab4ce498c3e90b9daa7425 P a16339bb0a5e4d3e93e3083c087b5fe7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 08:08:44.893896 21189 ts_tablet_manager.cc:1905] T 09456e56b7ab4ce498c3e90b9daa7425 P 509ae480ec1c4aacb55daff15185c729: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 08:08:44.896493 21189 ts_tablet_manager.cc:1918] T 09456e56b7ab4ce498c3e90b9daa7425 P 509ae480ec1c4aacb55daff15185c729: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250903 08:08:44.896600 21189 log.cc:1199] T 09456e56b7ab4ce498c3e90b9daa7425 P 509ae480ec1c4aacb55daff15185c729: Deleting WAL directory at /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-2/wal/wals/09456e56b7ab4ce498c3e90b9daa7425
I20250903 08:08:44.896586 21190 ts_tablet_manager.cc:1918] T 09456e56b7ab4ce498c3e90b9daa7425 P a16339bb0a5e4d3e93e3083c087b5fe7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250903 08:08:44.896673 21190 log.cc:1199] T 09456e56b7ab4ce498c3e90b9daa7425 P a16339bb0a5e4d3e93e3083c087b5fe7: Deleting WAL directory at /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-0/wal/wals/09456e56b7ab4ce498c3e90b9daa7425
I20250903 08:08:44.896925 21189 ts_tablet_manager.cc:1939] T 09456e56b7ab4ce498c3e90b9daa7425 P 509ae480ec1c4aacb55daff15185c729: Deleting consensus metadata
I20250903 08:08:44.897053 21190 ts_tablet_manager.cc:1939] T 09456e56b7ab4ce498c3e90b9daa7425 P a16339bb0a5e4d3e93e3083c087b5fe7: Deleting consensus metadata
I20250903 08:08:44.897814 13095 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUML90r/build/debug/bin/kudu with pid 19843
I20250903 08:08:44.899230 21191 tablet_replica.cc:331] T 09456e56b7ab4ce498c3e90b9daa7425 P 2b6d679b81794eac94e52a9454286dcd: stopping tablet replica
I20250903 08:08:44.899572 21191 raft_consensus.cc:2241] T 09456e56b7ab4ce498c3e90b9daa7425 P 2b6d679b81794eac94e52a9454286dcd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 08:08:44.899935 21191 raft_consensus.cc:2270] T 09456e56b7ab4ce498c3e90b9daa7425 P 2b6d679b81794eac94e52a9454286dcd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 08:08:44.902159 21191 ts_tablet_manager.cc:1905] T 09456e56b7ab4ce498c3e90b9daa7425 P 2b6d679b81794eac94e52a9454286dcd: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250903 08:08:44.905306 21022 connection.cc:537] server connection from 127.0.0.1:36654 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250903 08:08:44.905799 20752 connection.cc:537] server connection from 127.0.0.1:45866 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250903 08:08:44.906030 21191 ts_tablet_manager.cc:1918] T 09456e56b7ab4ce498c3e90b9daa7425 P 2b6d679b81794eac94e52a9454286dcd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250903 08:08:44.906127 21191 log.cc:1199] T 09456e56b7ab4ce498c3e90b9daa7425 P 2b6d679b81794eac94e52a9454286dcd: Deleting WAL directory at /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-1/wal/wals/09456e56b7ab4ce498c3e90b9daa7425
I20250903 08:08:44.906471 21191 ts_tablet_manager.cc:1939] T 09456e56b7ab4ce498c3e90b9daa7425 P 2b6d679b81794eac94e52a9454286dcd: Deleting consensus metadata
W20250903 08:08:44.907801 20889 connection.cc:441] server connection from 127.0.0.1:49014 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49014 (request call id 1) could send its response
I20250903 08:08:44.917256 21150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:44.908755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56946 (local address 127.12.201.252:35949)
0903 08:08:44.908893 (+ 138us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:44.908897 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:44.909161 (+ 264us) server_negotiation.cc:408] Connection header received
0903 08:08:44.909270 (+ 109us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:44.909275 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:44.909337 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:44.909421 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:44.910256 (+ 835us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.911116 (+ 860us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:44.911955 (+ 839us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.912158 (+ 203us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:44.913863 (+ 1705us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:44.913886 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:44.913890 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:08:44.913923 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:44.915555 (+ 1632us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:44.916145 (+ 590us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:44.916148 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:44.916149 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:44.916192 (+ 43us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:44.916434 (+ 242us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:44.916437 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:44.916440 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:44.916582 (+ 142us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:44.916711 (+ 129us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:44.916942 (+ 231us) server_negotiation.cc:300] Negotiation successful
0903 08:08:44.917055 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20250903 08:08:44.917335 21148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:44.908382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38768 (local address 127.12.201.253:42205)
0903 08:08:44.908525 (+ 143us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:44.908530 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:44.908764 (+ 234us) server_negotiation.cc:408] Connection header received
0903 08:08:44.908947 (+ 183us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:44.908950 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:44.909006 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:44.909059 (+ 53us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:44.910245 (+ 1186us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.910751 (+ 506us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:44.911563 (+ 812us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:44.911774 (+ 211us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:44.913577 (+ 1803us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:44.913598 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:44.913602 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:08:44.913628 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:44.915517 (+ 1889us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:44.916120 (+ 603us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:44.916125 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:44.916127 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:44.916175 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:44.916421 (+ 246us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:44.916425 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:44.916427 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:44.916605 (+ 178us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:44.916722 (+ 117us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:44.917045 (+ 323us) server_negotiation.cc:300] Negotiation successful
0903 08:08:44.917167 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20250903 08:08:45.879071 21199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:45.878640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:34143 (local address 127.12.201.195:50447)
0903 08:08:45.878884 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:45.878949 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.201.254:34143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":81,"threads_started":1}
W20250903 08:08:45.879385 21136 heartbeater.cc:646] Failed to heartbeat to 127.12.201.254:34143 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.12.201.254:34143: connect: Connection refused (error 111)
I20250903 08:08:45.881901 21200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:45.881490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:34143 (local address 127.12.201.194:42605)
0903 08:08:45.881722 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:45.881798 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.201.254:34143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":73,"threads_started":1}
W20250903 08:08:45.882282 21000 heartbeater.cc:646] Failed to heartbeat to 127.12.201.254:34143 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.12.201.254:34143: connect: Connection refused (error 111)
I20250903 08:08:45.884161 21201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:45.883773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:34143 (local address 127.12.201.193:42549)
0903 08:08:45.883980 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:45.884055 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.201.254:34143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":60,"threads_started":1}
W20250903 08:08:45.884408 20864 heartbeater.cc:646] Failed to heartbeat to 127.12.201.254:34143 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.12.201.254:34143: connect: Connection refused (error 111)
I20250903 08:08:46.391356 21202 raft_consensus.cc:491] T 00000000000000000000000000000000 P 094ea437f04c45fb93ad96244e3ca8a7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ff08db2a004746a6ab7e1d0fa91eed38)
I20250903 08:08:46.391552 21202 raft_consensus.cc:513] T 00000000000000000000000000000000 P 094ea437f04c45fb93ad96244e3ca8a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ff08db2a004746a6ab7e1d0fa91eed38" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 34143 } } peers { permanent_uuid: "094ea437f04c45fb93ad96244e3ca8a7" member_type: VOTER last_known_addr { host: "127.12.201.253" port: 42205 } attrs { promote: false } } peers { permanent_uuid: "0fba560828064814bed66061ed3ebc3d" member_type: VOTER last_known_addr { host: "127.12.201.252" port: 35949 } attrs { promote: false } }
I20250903 08:08:46.392470 21202 leader_election.cc:290] T 00000000000000000000000000000000 P 094ea437f04c45fb93ad96244e3ca8a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ff08db2a004746a6ab7e1d0fa91eed38 (127.12.201.254:34143), 0fba560828064814bed66061ed3ebc3d (127.12.201.252:35949)
I20250903 08:08:46.392942 20680 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "094ea437f04c45fb93ad96244e3ca8a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0fba560828064814bed66061ed3ebc3d" is_pre_election: true
I20250903 08:08:46.393010 21203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:46.392660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:34143 (local address 127.0.0.1:37648)
0903 08:08:46.392862 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:46.392925 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.201.254:34143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":57,"threads_started":1}
W20250903 08:08:46.393246 20471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.201.254:34143: connect: Connection refused (error 111)
I20250903 08:08:46.394006 21203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:46.393796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:34143 (local address 127.0.0.1:37650)
0903 08:08:46.393910 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:46.393950 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.201.254:34143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20250903 08:08:46.394131 20471 leader_election.cc:336] T 00000000000000000000000000000000 P 094ea437f04c45fb93ad96244e3ca8a7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ff08db2a004746a6ab7e1d0fa91eed38 (127.12.201.254:34143): Network error: Client connection negotiation failed: client connection to 127.12.201.254:34143: connect: Connection refused (error 111)
I20250903 08:08:46.394193 20471 leader_election.cc:304] T 00000000000000000000000000000000 P 094ea437f04c45fb93ad96244e3ca8a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 094ea437f04c45fb93ad96244e3ca8a7; no voters: 0fba560828064814bed66061ed3ebc3d, ff08db2a004746a6ab7e1d0fa91eed38
I20250903 08:08:46.394358 21202 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 094ea437f04c45fb93ad96244e3ca8a7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250903 08:08:46.396767 21205 raft_consensus.cc:491] T 00000000000000000000000000000000 P 0fba560828064814bed66061ed3ebc3d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ff08db2a004746a6ab7e1d0fa91eed38)
I20250903 08:08:46.396925 21205 raft_consensus.cc:513] T 00000000000000000000000000000000 P 0fba560828064814bed66061ed3ebc3d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ff08db2a004746a6ab7e1d0fa91eed38" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 34143 } } peers { permanent_uuid: "094ea437f04c45fb93ad96244e3ca8a7" member_type: VOTER last_known_addr { host: "127.12.201.253" port: 42205 } attrs { promote: false } } peers { permanent_uuid: "0fba560828064814bed66061ed3ebc3d" member_type: VOTER last_known_addr { host: "127.12.201.252" port: 35949 } attrs { promote: false } }
I20250903 08:08:46.397722 21205 leader_election.cc:290] T 00000000000000000000000000000000 P 0fba560828064814bed66061ed3ebc3d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ff08db2a004746a6ab7e1d0fa91eed38 (127.12.201.254:34143), 094ea437f04c45fb93ad96244e3ca8a7 (127.12.201.253:42205)
I20250903 08:08:46.398121 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0fba560828064814bed66061ed3ebc3d" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "094ea437f04c45fb93ad96244e3ca8a7" is_pre_election: true
I20250903 08:08:46.398187 21206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:46.397870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:34143 (local address 127.0.0.1:37656)
0903 08:08:46.398062 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:46.398115 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.201.254:34143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":52,"threads_started":1}
I20250903 08:08:46.398277 20497 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 094ea437f04c45fb93ad96244e3ca8a7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fba560828064814bed66061ed3ebc3d in term 1.
W20250903 08:08:46.398334 20654 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.201.254:34143: connect: Connection refused (error 111)
I20250903 08:08:46.398618 20657 leader_election.cc:304] T 00000000000000000000000000000000 P 0fba560828064814bed66061ed3ebc3d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 094ea437f04c45fb93ad96244e3ca8a7, 0fba560828064814bed66061ed3ebc3d; no voters:
I20250903 08:08:46.398813 21205 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 0fba560828064814bed66061ed3ebc3d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250903 08:08:46.398869 21205 raft_consensus.cc:491] T 00000000000000000000000000000000 P 0fba560828064814bed66061ed3ebc3d [term 1 FOLLOWER]: Starting leader election (detected failure of leader ff08db2a004746a6ab7e1d0fa91eed38)
I20250903 08:08:46.398897 21205 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 0fba560828064814bed66061ed3ebc3d [term 1 FOLLOWER]: Advancing to term 2
I20250903 08:08:46.399149 21206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:46.398884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:34143 (local address 127.0.0.1:37670)
0903 08:08:46.399056 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:46.399096 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.201.254:34143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20250903 08:08:46.399266 20654 leader_election.cc:336] T 00000000000000000000000000000000 P 0fba560828064814bed66061ed3ebc3d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ff08db2a004746a6ab7e1d0fa91eed38 (127.12.201.254:34143): Network error: Client connection negotiation failed: client connection to 127.12.201.254:34143: connect: Connection refused (error 111)
I20250903 08:08:46.400009 21205 raft_consensus.cc:513] T 00000000000000000000000000000000 P 0fba560828064814bed66061ed3ebc3d [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ff08db2a004746a6ab7e1d0fa91eed38" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 34143 } } peers { permanent_uuid: "094ea437f04c45fb93ad96244e3ca8a7" member_type: VOTER last_known_addr { host: "127.12.201.253" port: 42205 } attrs { promote: false } } peers { permanent_uuid: "0fba560828064814bed66061ed3ebc3d" member_type: VOTER last_known_addr { host: "127.12.201.252" port: 35949 } attrs { promote: false } }
I20250903 08:08:46.400333 21205 leader_election.cc:290] T 00000000000000000000000000000000 P 0fba560828064814bed66061ed3ebc3d [CANDIDATE]: Term 2 election: Requested vote from peers ff08db2a004746a6ab7e1d0fa91eed38 (127.12.201.254:34143), 094ea437f04c45fb93ad96244e3ca8a7 (127.12.201.253:42205)
I20250903 08:08:46.400655 21206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:46.400427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:34143 (local address 127.0.0.1:37680)
0903 08:08:46.400523 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:46.400595 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.201.254:34143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20250903 08:08:46.400671 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0fba560828064814bed66061ed3ebc3d" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "094ea437f04c45fb93ad96244e3ca8a7"
I20250903 08:08:46.400785 20497 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 094ea437f04c45fb93ad96244e3ca8a7 [term 1 FOLLOWER]: Advancing to term 2
I20250903 08:08:46.401240 21206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:46.401060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:34143 (local address 127.0.0.1:37694)
0903 08:08:46.401157 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:46.401198 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.201.254:34143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20250903 08:08:46.401330 20654 leader_election.cc:336] T 00000000000000000000000000000000 P 0fba560828064814bed66061ed3ebc3d [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ff08db2a004746a6ab7e1d0fa91eed38 (127.12.201.254:34143): Network error: Client connection negotiation failed: client connection to 127.12.201.254:34143: connect: Connection refused (error 111)
I20250903 08:08:46.401736 20497 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 094ea437f04c45fb93ad96244e3ca8a7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fba560828064814bed66061ed3ebc3d in term 2.
I20250903 08:08:46.402002 20657 leader_election.cc:304] T 00000000000000000000000000000000 P 0fba560828064814bed66061ed3ebc3d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 094ea437f04c45fb93ad96244e3ca8a7, 0fba560828064814bed66061ed3ebc3d; no voters: ff08db2a004746a6ab7e1d0fa91eed38
I20250903 08:08:46.402170 21205 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 0fba560828064814bed66061ed3ebc3d [term 2 FOLLOWER]: Leader election won for term 2
I20250903 08:08:46.402333 21205 raft_consensus.cc:695] T 00000000000000000000000000000000 P 0fba560828064814bed66061ed3ebc3d [term 2 LEADER]: Becoming Leader. State: Replica: 0fba560828064814bed66061ed3ebc3d, State: Running, Role: LEADER
I20250903 08:08:46.402559 21205 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0fba560828064814bed66061ed3ebc3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ff08db2a004746a6ab7e1d0fa91eed38" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 34143 } } peers { permanent_uuid: "094ea437f04c45fb93ad96244e3ca8a7" member_type: VOTER last_known_addr { host: "127.12.201.253" port: 42205 } attrs { promote: false } } peers { permanent_uuid: "0fba560828064814bed66061ed3ebc3d" member_type: VOTER last_known_addr { host: "127.12.201.252" port: 35949 } attrs { promote: false } }
I20250903 08:08:46.403779 21208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fba560828064814bed66061ed3ebc3d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0fba560828064814bed66061ed3ebc3d. Latest consensus state: current_term: 2 leader_uuid: "0fba560828064814bed66061ed3ebc3d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ff08db2a004746a6ab7e1d0fa91eed38" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 34143 } } peers { permanent_uuid: "094ea437f04c45fb93ad96244e3ca8a7" member_type: VOTER last_known_addr { host: "127.12.201.253" port: 42205 } attrs { promote: false } } peers { permanent_uuid: "0fba560828064814bed66061ed3ebc3d" member_type: VOTER last_known_addr { host: "127.12.201.252" port: 35949 } attrs { promote: false } } }
I20250903 08:08:46.403865 21208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fba560828064814bed66061ed3ebc3d [sys.catalog]: This master's current role is: LEADER
I20250903 08:08:46.404127 21210 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250903 08:08:46.406263 21210 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250903 08:08:46.406651 21210 catalog_manager.cc:1261] Loaded cluster ID: b7b764f24fd443ee92ab54013d2600ab
I20250903 08:08:46.406703 21210 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250903 08:08:46.407330 21210 catalog_manager.cc:1506] Loading token signing keys...
I20250903 08:08:46.407740 21210 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 0fba560828064814bed66061ed3ebc3d: Loaded TSK: 0
I20250903 08:08:46.408406 21210 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250903 08:08:46.425549 20668 catalog_manager.cc:2232] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56946:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 08:08:46.428344 20668 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 08:08:46.431730 21206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:46.431436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:34143 (local address 127.0.0.1:37712)
0903 08:08:46.431555 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:46.431620 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.201.254:34143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20250903 08:08:46.431800 20497 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 094ea437f04c45fb93ad96244e3ca8a7 [term 2 FOLLOWER]: Refusing update from remote peer 0fba560828064814bed66061ed3ebc3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20250903 08:08:46.432281 21205 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0fba560828064814bed66061ed3ebc3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "094ea437f04c45fb93ad96244e3ca8a7" member_type: VOTER last_known_addr { host: "127.12.201.253" port: 42205 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250903 08:08:46.432380 21206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:46.432178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:34143 (local address 127.0.0.1:37714)
0903 08:08:46.432279 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:46.432326 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.201.254:34143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20250903 08:08:46.432489 20654 consensus_peers.cc:489] T 00000000000000000000000000000000 P 0fba560828064814bed66061ed3ebc3d -> Peer ff08db2a004746a6ab7e1d0fa91eed38 (127.12.201.254:34143): Couldn't send request to peer ff08db2a004746a6ab7e1d0fa91eed38. Status: Network error: Client connection negotiation failed: client connection to 127.12.201.254:34143: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250903 08:08:46.433754 21202 sys_catalog.cc:455] T 00000000000000000000000000000000 P 094ea437f04c45fb93ad96244e3ca8a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0fba560828064814bed66061ed3ebc3d. Latest consensus state: current_term: 2 leader_uuid: "0fba560828064814bed66061ed3ebc3d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ff08db2a004746a6ab7e1d0fa91eed38" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 34143 } } peers { permanent_uuid: "094ea437f04c45fb93ad96244e3ca8a7" member_type: VOTER last_known_addr { host: "127.12.201.253" port: 42205 } attrs { promote: false } } peers { permanent_uuid: "0fba560828064814bed66061ed3ebc3d" member_type: VOTER last_known_addr { host: "127.12.201.252" port: 35949 } attrs { promote: false } } }
I20250903 08:08:46.433864 21202 sys_catalog.cc:458] T 00000000000000000000000000000000 P 094ea437f04c45fb93ad96244e3ca8a7 [sys.catalog]: This master's current role is: FOLLOWER
I20250903 08:08:46.434861 21205 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fba560828064814bed66061ed3ebc3d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "0fba560828064814bed66061ed3ebc3d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ff08db2a004746a6ab7e1d0fa91eed38" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 34143 } } peers { permanent_uuid: "094ea437f04c45fb93ad96244e3ca8a7" member_type: VOTER last_known_addr { host: "127.12.201.253" port: 42205 } attrs { promote: false } } peers { permanent_uuid: "0fba560828064814bed66061ed3ebc3d" member_type: VOTER last_known_addr { host: "127.12.201.252" port: 35949 } attrs { promote: false } } }
I20250903 08:08:46.434998 21205 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fba560828064814bed66061ed3ebc3d [sys.catalog]: This master's current role is: LEADER
I20250903 08:08:46.436143 21202 sys_catalog.cc:455] T 00000000000000000000000000000000 P 094ea437f04c45fb93ad96244e3ca8a7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "0fba560828064814bed66061ed3ebc3d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ff08db2a004746a6ab7e1d0fa91eed38" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 34143 } } peers { permanent_uuid: "094ea437f04c45fb93ad96244e3ca8a7" member_type: VOTER last_known_addr { host: "127.12.201.253" port: 42205 } attrs { promote: false } } peers { permanent_uuid: "0fba560828064814bed66061ed3ebc3d" member_type: VOTER last_known_addr { host: "127.12.201.252" port: 35949 } attrs { promote: false } } }
I20250903 08:08:46.436249 21202 sys_catalog.cc:458] T 00000000000000000000000000000000 P 094ea437f04c45fb93ad96244e3ca8a7 [sys.catalog]: This master's current role is: FOLLOWER
I20250903 08:08:46.443866 21206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:46.440692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.194:41585 (local address 127.0.0.1:49016)
0903 08:08:46.440877 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:46.440895 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:46.440986 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:46.441420 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:46.441423 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:46.441441 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 08:08:46.441670 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:46.441679 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:46.442652 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:46.442656 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:46.443482 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:46.443492 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:46.443600 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:46.443622 (+ 22us) client_negotiation.cc:769] Sending connection context
0903 08:08:46.443669 (+ 47us) client_negotiation.cc:240] Negotiation successful
0903 08:08:46.443737 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20250903 08:08:46.444772 21218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:46.441024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.193:43289 (local address 127.0.0.1:45882)
0903 08:08:46.441953 (+ 929us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:46.441964 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:46.442046 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:46.442387 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:46.442391 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:46.442406 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 08:08:46.442663 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:46.442670 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:46.443495 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:46.443498 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:46.444374 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:46.444383 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:46.444511 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:46.444529 (+ 18us) client_negotiation.cc:769] Sending connection context
0903 08:08:46.444582 (+ 53us) client_negotiation.cc:240] Negotiation successful
0903 08:08:46.444641 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":871,"thread_start_us":97,"threads_started":1}
I20250903 08:08:46.444815 21216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:46.440914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49016 (local address 127.12.201.194:41585)
0903 08:08:46.441189 (+ 275us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:46.441195 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:46.441210 (+ 15us) server_negotiation.cc:408] Connection header received
0903 08:08:46.441263 (+ 53us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:46.441266 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:46.441319 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:46.441410 (+ 91us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 08:08:46.441813 (+ 403us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:46.442526 (+ 713us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:46.444090 (+ 1564us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:46.444582 (+ 492us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:46.444614 (+ 32us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:46.444671 (+ 57us) server_negotiation.cc:300] Negotiation successful
0903 08:08:46.444719 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":84,"threads_started":1}
I20250903 08:08:46.445446 20933 tablet_service.cc:1468] Processing CreateTablet for tablet 0c67e101724045389db61fe0e01b0d08 (DEFAULT_TABLE table=test-table [id=cc019f61b92a41cbad6de36b3fad1705]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:08:46.445731 20933 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c67e101724045389db61fe0e01b0d08. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:08:46.446076 21217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:46.441316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.195:35143 (local address 127.0.0.1:36664)
0903 08:08:46.442244 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:46.442258 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:46.442325 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:46.442676 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:46.442679 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:46.442697 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 08:08:46.442947 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:46.442954 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:46.444921 (+ 1967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:46.444924 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:46.445708 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:46.445717 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:46.445835 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:46.445850 (+ 15us) client_negotiation.cc:769] Sending connection context
0903 08:08:46.445896 (+ 46us) client_negotiation.cc:240] Negotiation successful
0903 08:08:46.445942 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":850,"thread_start_us":77,"threads_started":1}
I20250903 08:08:46.446579 21219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:46.441456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36664 (local address 127.12.201.195:35143)
0903 08:08:46.441703 (+ 247us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:46.441709 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:46.442397 (+ 688us) server_negotiation.cc:408] Connection header received
0903 08:08:46.442450 (+ 53us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:46.442454 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:46.442504 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:46.442574 (+ 70us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 08:08:46.443074 (+ 500us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:46.443857 (+ 783us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:46.445841 (+ 1984us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:46.446335 (+ 494us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:46.446366 (+ 31us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:46.446409 (+ 43us) server_negotiation.cc:300] Negotiation successful
0903 08:08:46.446475 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":81,"threads_started":1}
I20250903 08:08:46.447047 21220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:46.441620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45882 (local address 127.12.201.193:43289)
0903 08:08:46.441830 (+ 210us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:46.441836 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:46.442028 (+ 192us) server_negotiation.cc:408] Connection header received
0903 08:08:46.442132 (+ 104us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:46.442137 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:46.442188 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:46.442279 (+ 91us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 08:08:46.442788 (+ 509us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:46.443377 (+ 589us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:46.446231 (+ 2854us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:46.446686 (+ 455us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:46.446728 (+ 42us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:46.446780 (+ 52us) server_negotiation.cc:300] Negotiation successful
0903 08:08:46.446855 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":53,"threads_started":1}
I20250903 08:08:46.447131 21069 tablet_service.cc:1468] Processing CreateTablet for tablet 0c67e101724045389db61fe0e01b0d08 (DEFAULT_TABLE table=test-table [id=cc019f61b92a41cbad6de36b3fad1705]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:08:46.447415 21069 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c67e101724045389db61fe0e01b0d08. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:08:46.447489 20797 tablet_service.cc:1468] Processing CreateTablet for tablet 0c67e101724045389db61fe0e01b0d08 (DEFAULT_TABLE table=test-table [id=cc019f61b92a41cbad6de36b3fad1705]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:08:46.447721 20797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c67e101724045389db61fe0e01b0d08. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:08:46.448303 21221 tablet_bootstrap.cc:492] T 0c67e101724045389db61fe0e01b0d08 P 2b6d679b81794eac94e52a9454286dcd: Bootstrap starting.
I20250903 08:08:46.449342 21221 tablet_bootstrap.cc:654] T 0c67e101724045389db61fe0e01b0d08 P 2b6d679b81794eac94e52a9454286dcd: Neither blocks nor log segments found. Creating new log.
I20250903 08:08:46.449589 21222 tablet_bootstrap.cc:492] T 0c67e101724045389db61fe0e01b0d08 P a16339bb0a5e4d3e93e3083c087b5fe7: Bootstrap starting.
I20250903 08:08:46.450309 21223 tablet_bootstrap.cc:492] T 0c67e101724045389db61fe0e01b0d08 P 509ae480ec1c4aacb55daff15185c729: Bootstrap starting.
I20250903 08:08:46.450765 21222 tablet_bootstrap.cc:654] T 0c67e101724045389db61fe0e01b0d08 P a16339bb0a5e4d3e93e3083c087b5fe7: Neither blocks nor log segments found. Creating new log.
I20250903 08:08:46.450878 21221 tablet_bootstrap.cc:492] T 0c67e101724045389db61fe0e01b0d08 P 2b6d679b81794eac94e52a9454286dcd: No bootstrap required, opened a new log
I20250903 08:08:46.450991 21221 ts_tablet_manager.cc:1397] T 0c67e101724045389db61fe0e01b0d08 P 2b6d679b81794eac94e52a9454286dcd: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250903 08:08:46.451344 21223 tablet_bootstrap.cc:654] T 0c67e101724045389db61fe0e01b0d08 P 509ae480ec1c4aacb55daff15185c729: Neither blocks nor log segments found. Creating new log.
I20250903 08:08:46.451494 21221 raft_consensus.cc:357] T 0c67e101724045389db61fe0e01b0d08 P 2b6d679b81794eac94e52a9454286dcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b6d679b81794eac94e52a9454286dcd" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 41585 } } peers { permanent_uuid: "a16339bb0a5e4d3e93e3083c087b5fe7" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 43289 } } peers { permanent_uuid: "509ae480ec1c4aacb55daff15185c729" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 35143 } }
I20250903 08:08:46.451614 21221 raft_consensus.cc:383] T 0c67e101724045389db61fe0e01b0d08 P 2b6d679b81794eac94e52a9454286dcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:08:46.451661 21221 raft_consensus.cc:738] T 0c67e101724045389db61fe0e01b0d08 P 2b6d679b81794eac94e52a9454286dcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b6d679b81794eac94e52a9454286dcd, State: Initialized, Role: FOLLOWER
I20250903 08:08:46.451792 21221 consensus_queue.cc:260] T 0c67e101724045389db61fe0e01b0d08 P 2b6d679b81794eac94e52a9454286dcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b6d679b81794eac94e52a9454286dcd" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 41585 } } peers { permanent_uuid: "a16339bb0a5e4d3e93e3083c087b5fe7" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 43289 } } peers { permanent_uuid: "509ae480ec1c4aacb55daff15185c729" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 35143 } }
I20250903 08:08:46.451911 21222 tablet_bootstrap.cc:492] T 0c67e101724045389db61fe0e01b0d08 P a16339bb0a5e4d3e93e3083c087b5fe7: No bootstrap required, opened a new log
I20250903 08:08:46.452005 21222 ts_tablet_manager.cc:1397] T 0c67e101724045389db61fe0e01b0d08 P a16339bb0a5e4d3e93e3083c087b5fe7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250903 08:08:46.452147 21221 ts_tablet_manager.cc:1428] T 0c67e101724045389db61fe0e01b0d08 P 2b6d679b81794eac94e52a9454286dcd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250903 08:08:46.452417 21222 raft_consensus.cc:357] T 0c67e101724045389db61fe0e01b0d08 P a16339bb0a5e4d3e93e3083c087b5fe7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b6d679b81794eac94e52a9454286dcd" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 41585 } } peers { permanent_uuid: "a16339bb0a5e4d3e93e3083c087b5fe7" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 43289 } } peers { permanent_uuid: "509ae480ec1c4aacb55daff15185c729" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 35143 } }
I20250903 08:08:46.452507 21223 tablet_bootstrap.cc:492] T 0c67e101724045389db61fe0e01b0d08 P 509ae480ec1c4aacb55daff15185c729: No bootstrap required, opened a new log
I20250903 08:08:46.452507 21222 raft_consensus.cc:383] T 0c67e101724045389db61fe0e01b0d08 P a16339bb0a5e4d3e93e3083c087b5fe7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:08:46.452548 21222 raft_consensus.cc:738] T 0c67e101724045389db61fe0e01b0d08 P a16339bb0a5e4d3e93e3083c087b5fe7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a16339bb0a5e4d3e93e3083c087b5fe7, State: Initialized, Role: FOLLOWER
I20250903 08:08:46.452586 21223 ts_tablet_manager.cc:1397] T 0c67e101724045389db61fe0e01b0d08 P 509ae480ec1c4aacb55daff15185c729: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250903 08:08:46.452682 21222 consensus_queue.cc:260] T 0c67e101724045389db61fe0e01b0d08 P a16339bb0a5e4d3e93e3083c087b5fe7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b6d679b81794eac94e52a9454286dcd" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 41585 } } peers { permanent_uuid: "a16339bb0a5e4d3e93e3083c087b5fe7" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 43289 } } peers { permanent_uuid: "509ae480ec1c4aacb55daff15185c729" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 35143 } }
I20250903 08:08:46.452967 21222 ts_tablet_manager.cc:1428] T 0c67e101724045389db61fe0e01b0d08 P a16339bb0a5e4d3e93e3083c087b5fe7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250903 08:08:46.453718 21230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:46.453341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:34143 (local address 127.12.201.193:37683)
0903 08:08:46.453563 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:46.453616 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.201.254:34143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":98,"threads_started":1}
I20250903 08:08:46.453939 21229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:46.453248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:34143 (local address 127.12.201.194:38653)
0903 08:08:46.453790 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:46.453854 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.201.254:34143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":91,"threads_started":1}
I20250903 08:08:46.454649 21230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:46.454439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:34143 (local address 127.12.201.193:55499)
0903 08:08:46.454536 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:46.454585 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.201.254:34143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20250903 08:08:46.454710 21229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:46.454364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:34143 (local address 127.12.201.194:53701)
0903 08:08:46.454603 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:46.454651 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.201.254:34143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164}
I20250903 08:08:46.455768 20999 heartbeater.cc:499] Master 127.12.201.252:35949 was elected leader, sending a full tablet report...
I20250903 08:08:46.456179 20863 heartbeater.cc:499] Master 127.12.201.252:35949 was elected leader, sending a full tablet report...
I20250903 08:08:46.452996 21223 raft_consensus.cc:357] T 0c67e101724045389db61fe0e01b0d08 P 509ae480ec1c4aacb55daff15185c729 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b6d679b81794eac94e52a9454286dcd" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 41585 } } peers { permanent_uuid: "a16339bb0a5e4d3e93e3083c087b5fe7" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 43289 } } peers { permanent_uuid: "509ae480ec1c4aacb55daff15185c729" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 35143 } }
I20250903 08:08:46.457123 21223 raft_consensus.cc:383] T 0c67e101724045389db61fe0e01b0d08 P 509ae480ec1c4aacb55daff15185c729 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:08:46.457181 21223 raft_consensus.cc:738] T 0c67e101724045389db61fe0e01b0d08 P 509ae480ec1c4aacb55daff15185c729 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 509ae480ec1c4aacb55daff15185c729, State: Initialized, Role: FOLLOWER
I20250903 08:08:46.457347 21223 consensus_queue.cc:260] T 0c67e101724045389db61fe0e01b0d08 P 509ae480ec1c4aacb55daff15185c729 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b6d679b81794eac94e52a9454286dcd" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 41585 } } peers { permanent_uuid: "a16339bb0a5e4d3e93e3083c087b5fe7" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 43289 } } peers { permanent_uuid: "509ae480ec1c4aacb55daff15185c729" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 35143 } }
I20250903 08:08:46.457692 21223 ts_tablet_manager.cc:1428] T 0c67e101724045389db61fe0e01b0d08 P 509ae480ec1c4aacb55daff15185c729: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20250903 08:08:46.458658 21232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:46.457979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:34143 (local address 127.12.201.195:43075)
0903 08:08:46.458501 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:46.458555 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.201.254:34143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":108,"threads_started":1}
I20250903 08:08:46.459170 21135 heartbeater.cc:499] Master 127.12.201.252:35949 was elected leader, sending a full tablet report...
I20250903 08:08:46.459326 21232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:46.459093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:34143 (local address 127.12.201.195:49005)
0903 08:08:46.459190 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:46.459257 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.201.254:34143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20250903 08:08:46.471827 21002 tablet.cc:2378] T 0c67e101724045389db61fe0e01b0d08 P 2b6d679b81794eac94e52a9454286dcd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250903 08:08:46.565686 21227 raft_consensus.cc:491] T 0c67e101724045389db61fe0e01b0d08 P 2b6d679b81794eac94e52a9454286dcd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 08:08:46.565834 21227 raft_consensus.cc:513] T 0c67e101724045389db61fe0e01b0d08 P 2b6d679b81794eac94e52a9454286dcd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b6d679b81794eac94e52a9454286dcd" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 41585 } } peers { permanent_uuid: "a16339bb0a5e4d3e93e3083c087b5fe7" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 43289 } } peers { permanent_uuid: "509ae480ec1c4aacb55daff15185c729" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 35143 } }
I20250903 08:08:46.566845 21227 leader_election.cc:290] T 0c67e101724045389db61fe0e01b0d08 P 2b6d679b81794eac94e52a9454286dcd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a16339bb0a5e4d3e93e3083c087b5fe7 (127.12.201.193:43289), 509ae480ec1c4aacb55daff15185c729 (127.12.201.195:35143)
I20250903 08:08:46.569700 21229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:46.567088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.193:43289 (local address 127.12.201.194:60035)
0903 08:08:46.567218 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:46.567238 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:46.567350 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:46.567707 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:46.567709 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:46.567725 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 08:08:46.568024 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:46.568033 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:46.568738 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:46.568742 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:46.569313 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:46.569320 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:46.569430 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:46.569452 (+ 22us) client_negotiation.cc:769] Sending connection context
0903 08:08:46.569504 (+ 52us) client_negotiation.cc:240] Negotiation successful
0903 08:08:46.569560 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20250903 08:08:46.569890 21233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:46.567387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.195:35143 (local address 127.12.201.194:59411)
0903 08:08:46.567686 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:46.567698 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:46.567785 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:46.568011 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:46.568013 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:46.568027 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 08:08:46.568240 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:46.568245 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:46.568950 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:46.568953 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:46.569497 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:46.569504 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:46.569609 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:46.569628 (+ 19us) client_negotiation.cc:769] Sending connection context
0903 08:08:46.569678 (+ 50us) client_negotiation.cc:240] Negotiation successful
0903 08:08:46.569731 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":132,"threads_started":1}
I20250903 08:08:46.570231 21220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:46.567187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.194:60035 (local address 127.12.201.193:43289)
0903 08:08:46.567336 (+ 149us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:46.567341 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:46.567357 (+ 16us) server_negotiation.cc:408] Connection header received
0903 08:08:46.567446 (+ 89us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:46.567449 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:46.567494 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:46.567562 (+ 68us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 08:08:46.568157 (+ 595us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:46.568632 (+ 475us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:46.569434 (+ 802us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:46.569932 (+ 498us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:46.569976 (+ 44us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:46.570035 (+ 59us) server_negotiation.cc:300] Negotiation successful
0903 08:08:46.570102 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20250903 08:08:46.570397 21219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:46.567505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.194:59411 (local address 127.12.201.195:35143)
0903 08:08:46.567652 (+ 147us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:46.567658 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:46.567772 (+ 114us) server_negotiation.cc:408] Connection header received
0903 08:08:46.567865 (+ 93us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:46.567869 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:46.567917 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:46.567991 (+ 74us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 08:08:46.568361 (+ 370us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:46.568832 (+ 471us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:46.569607 (+ 775us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:46.570131 (+ 524us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:46.570166 (+ 35us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:46.570215 (+ 49us) server_negotiation.cc:300] Negotiation successful
0903 08:08:46.570271 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20250903 08:08:46.570705 20817 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c67e101724045389db61fe0e01b0d08" candidate_uuid: "2b6d679b81794eac94e52a9454286dcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a16339bb0a5e4d3e93e3083c087b5fe7" is_pre_election: true
I20250903 08:08:46.570868 21089 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c67e101724045389db61fe0e01b0d08" candidate_uuid: "2b6d679b81794eac94e52a9454286dcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "509ae480ec1c4aacb55daff15185c729" is_pre_election: true
I20250903 08:08:46.570931 20817 raft_consensus.cc:2466] T 0c67e101724045389db61fe0e01b0d08 P a16339bb0a5e4d3e93e3083c087b5fe7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b6d679b81794eac94e52a9454286dcd in term 0.
I20250903 08:08:46.571023 21089 raft_consensus.cc:2466] T 0c67e101724045389db61fe0e01b0d08 P 509ae480ec1c4aacb55daff15185c729 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b6d679b81794eac94e52a9454286dcd in term 0.
I20250903 08:08:46.571360 20887 leader_election.cc:304] T 0c67e101724045389db61fe0e01b0d08 P 2b6d679b81794eac94e52a9454286dcd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b6d679b81794eac94e52a9454286dcd, 509ae480ec1c4aacb55daff15185c729; no voters:
I20250903 08:08:46.571586 21227 raft_consensus.cc:2802] T 0c67e101724045389db61fe0e01b0d08 P 2b6d679b81794eac94e52a9454286dcd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 08:08:46.571647 21227 raft_consensus.cc:491] T 0c67e101724045389db61fe0e01b0d08 P 2b6d679b81794eac94e52a9454286dcd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 08:08:46.571691 21227 raft_consensus.cc:3058] T 0c67e101724045389db61fe0e01b0d08 P 2b6d679b81794eac94e52a9454286dcd [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:08:46.572594 21227 raft_consensus.cc:513] T 0c67e101724045389db61fe0e01b0d08 P 2b6d679b81794eac94e52a9454286dcd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b6d679b81794eac94e52a9454286dcd" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 41585 } } peers { permanent_uuid: "a16339bb0a5e4d3e93e3083c087b5fe7" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 43289 } } peers { permanent_uuid: "509ae480ec1c4aacb55daff15185c729" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 35143 } }
I20250903 08:08:46.572938 21227 leader_election.cc:290] T 0c67e101724045389db61fe0e01b0d08 P 2b6d679b81794eac94e52a9454286dcd [CANDIDATE]: Term 1 election: Requested vote from peers a16339bb0a5e4d3e93e3083c087b5fe7 (127.12.201.193:43289), 509ae480ec1c4aacb55daff15185c729 (127.12.201.195:35143)
I20250903 08:08:46.573289 20817 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c67e101724045389db61fe0e01b0d08" candidate_uuid: "2b6d679b81794eac94e52a9454286dcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a16339bb0a5e4d3e93e3083c087b5fe7"
I20250903 08:08:46.573349 21089 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c67e101724045389db61fe0e01b0d08" candidate_uuid: "2b6d679b81794eac94e52a9454286dcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "509ae480ec1c4aacb55daff15185c729"
I20250903 08:08:46.573414 20817 raft_consensus.cc:3058] T 0c67e101724045389db61fe0e01b0d08 P a16339bb0a5e4d3e93e3083c087b5fe7 [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:08:46.573460 21089 raft_consensus.cc:3058] T 0c67e101724045389db61fe0e01b0d08 P 509ae480ec1c4aacb55daff15185c729 [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:08:46.574502 21089 raft_consensus.cc:2466] T 0c67e101724045389db61fe0e01b0d08 P 509ae480ec1c4aacb55daff15185c729 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b6d679b81794eac94e52a9454286dcd in term 1.
I20250903 08:08:46.574502 20817 raft_consensus.cc:2466] T 0c67e101724045389db61fe0e01b0d08 P a16339bb0a5e4d3e93e3083c087b5fe7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b6d679b81794eac94e52a9454286dcd in term 1.
I20250903 08:08:46.574800 20887 leader_election.cc:304] T 0c67e101724045389db61fe0e01b0d08 P 2b6d679b81794eac94e52a9454286dcd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b6d679b81794eac94e52a9454286dcd, a16339bb0a5e4d3e93e3083c087b5fe7; no voters:
I20250903 08:08:46.574959 21227 raft_consensus.cc:2802] T 0c67e101724045389db61fe0e01b0d08 P 2b6d679b81794eac94e52a9454286dcd [term 1 FOLLOWER]: Leader election won for term 1
I20250903 08:08:46.575191 21227 raft_consensus.cc:695] T 0c67e101724045389db61fe0e01b0d08 P 2b6d679b81794eac94e52a9454286dcd [term 1 LEADER]: Becoming Leader. State: Replica: 2b6d679b81794eac94e52a9454286dcd, State: Running, Role: LEADER
I20250903 08:08:46.575435 21227 consensus_queue.cc:237] T 0c67e101724045389db61fe0e01b0d08 P 2b6d679b81794eac94e52a9454286dcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b6d679b81794eac94e52a9454286dcd" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 41585 } } peers { permanent_uuid: "a16339bb0a5e4d3e93e3083c087b5fe7" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 43289 } } peers { permanent_uuid: "509ae480ec1c4aacb55daff15185c729" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 35143 } }
I20250903 08:08:46.575892 21233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:46.575660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:34143 (local address 127.12.201.194:35089)
0903 08:08:46.575773 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:46.575833 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.201.254:34143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20250903 08:08:46.577744 20668 catalog_manager.cc:5582] T 0c67e101724045389db61fe0e01b0d08 P 2b6d679b81794eac94e52a9454286dcd reported cstate change: term changed from 0 to 1, leader changed from <none> to 2b6d679b81794eac94e52a9454286dcd (127.12.201.194). New cstate: current_term: 1 leader_uuid: "2b6d679b81794eac94e52a9454286dcd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b6d679b81794eac94e52a9454286dcd" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 41585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a16339bb0a5e4d3e93e3083c087b5fe7" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 43289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "509ae480ec1c4aacb55daff15185c729" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 35143 } health_report { overall_health: UNKNOWN } } }
I20250903 08:08:46.601169 21216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:46.598077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49032 (local address 127.12.201.194:41585)
0903 08:08:46.598231 (+ 154us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:46.598238 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:46.598253 (+ 15us) server_negotiation.cc:408] Connection header received
0903 08:08:46.598336 (+ 83us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:46.598338 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:46.598391 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:46.598480 (+ 89us) server_negotiation.cc:227] Negotiated authn=TOKEN
0903 08:08:46.598815 (+ 335us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:46.599364 (+ 549us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:46.600014 (+ 650us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:46.600210 (+ 196us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:46.600265 (+ 55us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0903 08:08:46.600734 (+ 469us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0903 08:08:46.600841 (+ 107us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:46.600963 (+ 122us) server_negotiation.cc:300] Negotiation successful
0903 08:08:46.601014 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20250903 08:08:46.603935 21089 raft_consensus.cc:1273] T 0c67e101724045389db61fe0e01b0d08 P 509ae480ec1c4aacb55daff15185c729 [term 1 FOLLOWER]: Refusing update from remote peer 2b6d679b81794eac94e52a9454286dcd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 08:08:46.604241 20817 raft_consensus.cc:1273] T 0c67e101724045389db61fe0e01b0d08 P a16339bb0a5e4d3e93e3083c087b5fe7 [term 1 FOLLOWER]: Refusing update from remote peer 2b6d679b81794eac94e52a9454286dcd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 08:08:46.604446 21227 consensus_queue.cc:1035] T 0c67e101724045389db61fe0e01b0d08 P 2b6d679b81794eac94e52a9454286dcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "509ae480ec1c4aacb55daff15185c729" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 35143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 08:08:46.604774 21234 consensus_queue.cc:1035] T 0c67e101724045389db61fe0e01b0d08 P 2b6d679b81794eac94e52a9454286dcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a16339bb0a5e4d3e93e3083c087b5fe7" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 43289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 08:08:46.606281 21232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:46.605963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:34143 (local address 127.12.201.195:37275)
0903 08:08:46.606138 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:46.606200 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.201.254:34143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20250903 08:08:46.606595 21230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:46.606320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:34143 (local address 127.12.201.193:40725)
0903 08:08:46.606467 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:46.606515 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.201.254:34143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20250903 08:08:46.609956 21233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:46.609689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:34143 (local address 127.12.201.194:56593)
0903 08:08:46.609832 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:46.609882 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.201.254:34143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20250903 08:08:46.611409 21232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:46.611158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:34143 (local address 127.12.201.195:38789)
0903 08:08:46.611299 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:46.611338 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.201.254:34143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20250903 08:08:46.613139 21230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:46.612873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:34143 (local address 127.12.201.193:40643)
0903 08:08:46.613001 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:46.613061 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.201.254:34143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20250903 08:08:46.614681 21233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:46.614419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:34143 (local address 127.12.201.194:59027)
0903 08:08:46.614550 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:46.614606 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.201.254:34143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20250903 08:08:46.618217 20669 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56946:
table { table_name: "test-table" } modify_external_catalogs: true
I20250903 08:08:46.618366 20669 catalog_manager.cc:2730] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56946:
table { table_name: "test-table" } modify_external_catalogs: true
I20250903 08:08:46.622007 20669 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 0fba560828064814bed66061ed3ebc3d: Sending DeleteTablet for 3 replicas of tablet 0c67e101724045389db61fe0e01b0d08
I20250903 08:08:46.622946 20797 tablet_service.cc:1515] Processing DeleteTablet for tablet 0c67e101724045389db61fe0e01b0d08 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 08:08:46 UTC) from {username='oryx', principal='oryx/127.12.201.252@KRBTEST.COM'} at 127.0.0.1:45882
I20250903 08:08:46.622959 21069 tablet_service.cc:1515] Processing DeleteTablet for tablet 0c67e101724045389db61fe0e01b0d08 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 08:08:46 UTC) from {username='oryx', principal='oryx/127.12.201.252@KRBTEST.COM'} at 127.0.0.1:36664
I20250903 08:08:46.623603 20933 tablet_service.cc:1515] Processing DeleteTablet for tablet 0c67e101724045389db61fe0e01b0d08 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 08:08:46 UTC) from {username='oryx', principal='oryx/127.12.201.252@KRBTEST.COM'} at 127.0.0.1:49016
I20250903 08:08:46.623674 13095 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUML90r/build/debug/bin/kudu with pid 20734
I20250903 08:08:46.623689 21248 tablet_replica.cc:331] T 0c67e101724045389db61fe0e01b0d08 P a16339bb0a5e4d3e93e3083c087b5fe7: stopping tablet replica
I20250903 08:08:46.625123 21249 tablet_replica.cc:331] T 0c67e101724045389db61fe0e01b0d08 P 509ae480ec1c4aacb55daff15185c729: stopping tablet replica
I20250903 08:08:46.625250 21249 raft_consensus.cc:2241] T 0c67e101724045389db61fe0e01b0d08 P 509ae480ec1c4aacb55daff15185c729 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 08:08:46.625392 21249 raft_consensus.cc:2270] T 0c67e101724045389db61fe0e01b0d08 P 509ae480ec1c4aacb55daff15185c729 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 08:08:46.625674 21250 tablet_replica.cc:331] T 0c67e101724045389db61fe0e01b0d08 P 2b6d679b81794eac94e52a9454286dcd: stopping tablet replica
I20250903 08:08:46.625802 21250 raft_consensus.cc:2241] T 0c67e101724045389db61fe0e01b0d08 P 2b6d679b81794eac94e52a9454286dcd [term 1 LEADER]: Raft consensus shutting down.
I20250903 08:08:46.626134 21250 raft_consensus.cc:2270] T 0c67e101724045389db61fe0e01b0d08 P 2b6d679b81794eac94e52a9454286dcd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 08:08:46.626471 21249 ts_tablet_manager.cc:1905] T 0c67e101724045389db61fe0e01b0d08 P 509ae480ec1c4aacb55daff15185c729: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 08:08:46.626788 21250 ts_tablet_manager.cc:1905] T 0c67e101724045389db61fe0e01b0d08 P 2b6d679b81794eac94e52a9454286dcd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 08:08:46.629146 21250 ts_tablet_manager.cc:1918] T 0c67e101724045389db61fe0e01b0d08 P 2b6d679b81794eac94e52a9454286dcd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250903 08:08:46.629236 21250 log.cc:1199] T 0c67e101724045389db61fe0e01b0d08 P 2b6d679b81794eac94e52a9454286dcd: Deleting WAL directory at /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-1/wal/wals/0c67e101724045389db61fe0e01b0d08
I20250903 08:08:46.629549 21250 ts_tablet_manager.cc:1939] T 0c67e101724045389db61fe0e01b0d08 P 2b6d679b81794eac94e52a9454286dcd: Deleting consensus metadata
I20250903 08:08:46.629535 21249 ts_tablet_manager.cc:1918] T 0c67e101724045389db61fe0e01b0d08 P 509ae480ec1c4aacb55daff15185c729: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250903 08:08:46.629695 21249 log.cc:1199] T 0c67e101724045389db61fe0e01b0d08 P 509ae480ec1c4aacb55daff15185c729: Deleting WAL directory at /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756886860253814-13095-0/minicluster-data/ts-2/wal/wals/0c67e101724045389db61fe0e01b0d08
I20250903 08:08:46.630043 21249 ts_tablet_manager.cc:1939] T 0c67e101724045389db61fe0e01b0d08 P 509ae480ec1c4aacb55daff15185c729: Deleting consensus metadata
I20250903 08:08:46.630357 20654 catalog_manager.cc:4928] TS 2b6d679b81794eac94e52a9454286dcd (127.12.201.194:41585): tablet 0c67e101724045389db61fe0e01b0d08 (table test-table [id=cc019f61b92a41cbad6de36b3fad1705]) successfully deleted
I20250903 08:08:46.630817 20655 catalog_manager.cc:4928] TS 509ae480ec1c4aacb55daff15185c729 (127.12.201.195:35143): tablet 0c67e101724045389db61fe0e01b0d08 (table test-table [id=cc019f61b92a41cbad6de36b3fad1705]) successfully deleted
W20250903 08:08:46.635362 21217 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 08:08:46.635074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.193:43289 (local address 127.0.0.1:45894)
0903 08:08:46.635199 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:46.635245 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.201.193:43289: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":36}
I20250903 08:08:46.635375 13095 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUML90r/build/debug/bin/kudu with pid 20870
W20250903 08:08:46.638134 20655 catalog_manager.cc:4671] TS a16339bb0a5e4d3e93e3083c087b5fe7 (127.12.201.193:43289): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0c67e101724045389db61fe0e01b0d08: Network error: Client connection negotiation failed: client connection to 127.12.201.193:43289: connect: Connection reset by peer (error 104)
I20250903 08:08:46.643035 13095 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUML90r/build/debug/bin/kudu with pid 21006
I20250903 08:08:46.648862 13095 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUML90r/build/debug/bin/kudu with pid 20347
I20250903 08:08:46.654747 13095 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUML90r/build/debug/bin/kudu with pid 20459
2025-09-03T08:08:46Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7372 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 03 08:08:46 dist-test-slave-27r9 krb5kdc[21253](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 03 08:08:46 dist-test-slave-27r9 krb5kdc[21253](info): set up 2 sockets
Sep 03 08:08:46 dist-test-slave-27r9 krb5kdc[21253](info): commencing operation
krb5kdc: starting...
W20250903 08:08:48.643682 13095 mini_kdc.cc:121] Time spent starting KDC: real 1.965s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/krb5kdc/test-user.keytab.
Sep 03 08:08:48 dist-test-slave-27r9 krb5kdc[21253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-09-03T08:08:48Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-03T08:08:48Z Disabled control of system clock
WARNING: no policy specified for oryx/127.12.201.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.201.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.201.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.12.201.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.12.201.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.201.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.201.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.12.201.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/krb5kdc/oryx.keytab.
I20250903 08:08:48.776271 13095 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUML90r/build/debug/bin/kudu
/tmp/dist-test-taskUML90r/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.201.254:38257
--webserver_interface=127.12.201.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/krb5kdc/oryx.keytab
--principal=oryx/127.12.201.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.201.212:42635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.201.254:38257
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 08:08:48.882694 21269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:08:48.882936 21269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:08:48.883034 21269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:08:48.886235 21269 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250903 08:08:48.886288 21269 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20250903 08:08:48.886309 21269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 08:08:48.886348 21269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:08:48.886391 21269 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250903 08:08:48.886437 21269 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250903 08:08:48.890619 21269 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.201.212:42635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.201.254:38257
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/krb5kdc/oryx.keytab
--principal=oryx/127.12.201.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.201.254:38257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.201.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21269
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 07:50:08 UTC on 5fd53c4cbb9d
build id 7874
I20250903 08:08:48.891659 21269 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:08:48.892525 21269 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:08:48.897879 21277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:48.897866 21274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:48.897866 21275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:08:48.898305 21269 server_base.cc:1047] running on GCE node
I20250903 08:08:48.898790 21269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:08:48.899693 21269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:08:48.900846 21269 hybrid_clock.cc:648] HybridClock initialized: now 1756886928900824 us; error 36 us; skew 500 ppm
Sep 03 08:08:48 dist-test-slave-27r9 krb5kdc[21253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886928, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.201.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 08:08:48.903541 21269 init.cc:377] Logged in from keytab as oryx/127.12.201.254@KRBTEST.COM (short username oryx)
I20250903 08:08:48.904549 21269 webserver.cc:480] Webserver started at http://127.12.201.254:39147/ using document root <none> and password file <none>
I20250903 08:08:48.905081 21269 fs_manager.cc:362] Metadata directory not provided
I20250903 08:08:48.905146 21269 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:08:48.905326 21269 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:08:48.907059 21269 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/minicluster-data/master-0/data/instance:
uuid: "3aa107cc15d641bca7f21dd4251fdf15"
format_stamp: "Formatted at 2025-09-03 08:08:48 on dist-test-slave-27r9"
server_key: "73d9b5877fc282ef24a7b290add538ce"
server_key_iv: "3e1d70ef074c384c99fc76c3c35574ad"
server_key_version: "encryptionkey@0"
I20250903 08:08:48.907531 21269 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/minicluster-data/master-0/wal/instance:
uuid: "3aa107cc15d641bca7f21dd4251fdf15"
format_stamp: "Formatted at 2025-09-03 08:08:48 on dist-test-slave-27r9"
server_key: "73d9b5877fc282ef24a7b290add538ce"
server_key_iv: "3e1d70ef074c384c99fc76c3c35574ad"
server_key_version: "encryptionkey@0"
I20250903 08:08:48.910953 21269 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20250903 08:08:48.913115 21284 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:48.914069 21269 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250903 08:08:48.914179 21269 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/minicluster-data/master-0/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/minicluster-data/master-0/wal
uuid: "3aa107cc15d641bca7f21dd4251fdf15"
format_stamp: "Formatted at 2025-09-03 08:08:48 on dist-test-slave-27r9"
server_key: "73d9b5877fc282ef24a7b290add538ce"
server_key_iv: "3e1d70ef074c384c99fc76c3c35574ad"
server_key_version: "encryptionkey@0"
I20250903 08:08:48.914268 21269 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:08:48.925115 21269 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:08:48.927745 21269 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:08:48.927920 21269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:08:48.936017 21346 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.201.254:38257 every 8 connection(s)
I20250903 08:08:48.936021 21269 rpc_server.cc:307] RPC server started. Bound to: 127.12.201.254:38257
I20250903 08:08:48.937098 21269 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/minicluster-data/master-0/data/info.pb
I20250903 08:08:48.939617 21347 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:08:48.941603 13095 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUML90r/build/debug/bin/kudu as pid 21269
I20250903 08:08:48.941718 13095 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/minicluster-data/master-0/wal/instance
I20250903 08:08:48.941946 13095 external_mini_cluster.cc:1468] Setting key 59f39fad55e8a8c50e8d98ba87ff12e4
I20250903 08:08:48.945573 21347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3aa107cc15d641bca7f21dd4251fdf15: Bootstrap starting.
I20250903 08:08:48.947765 21347 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3aa107cc15d641bca7f21dd4251fdf15: Neither blocks nor log segments found. Creating new log.
I20250903 08:08:48.948503 21347 log.cc:826] T 00000000000000000000000000000000 P 3aa107cc15d641bca7f21dd4251fdf15: Log is configured to *not* fsync() on all Append() calls
Sep 03 08:08:48 dist-test-slave-27r9 krb5kdc[21253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM
I20250903 08:08:48.950377 21347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3aa107cc15d641bca7f21dd4251fdf15: No bootstrap required, opened a new log
I20250903 08:08:48.953071 21347 raft_consensus.cc:357] T 00000000000000000000000000000000 P 3aa107cc15d641bca7f21dd4251fdf15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aa107cc15d641bca7f21dd4251fdf15" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 38257 } }
I20250903 08:08:48.953274 21347 raft_consensus.cc:383] T 00000000000000000000000000000000 P 3aa107cc15d641bca7f21dd4251fdf15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:08:48.953334 21347 raft_consensus.cc:738] T 00000000000000000000000000000000 P 3aa107cc15d641bca7f21dd4251fdf15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aa107cc15d641bca7f21dd4251fdf15, State: Initialized, Role: FOLLOWER
I20250903 08:08:48.953765 21347 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3aa107cc15d641bca7f21dd4251fdf15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aa107cc15d641bca7f21dd4251fdf15" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 38257 } }
I20250903 08:08:48.953892 21347 raft_consensus.cc:397] T 00000000000000000000000000000000 P 3aa107cc15d641bca7f21dd4251fdf15 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250903 08:08:48.953958 21347 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3aa107cc15d641bca7f21dd4251fdf15 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250903 08:08:48.954092 21347 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 3aa107cc15d641bca7f21dd4251fdf15 [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:08:48.954133 21350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:48.943147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45642 (local address 127.12.201.254:38257)
0903 08:08:48.943619 (+ 472us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:48.943628 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:48.943663 (+ 35us) server_negotiation.cc:408] Connection header received
0903 08:08:48.944208 (+ 545us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:48.944225 (+ 17us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:48.944514 (+ 289us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:48.944813 (+ 299us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:48.945774 (+ 961us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:48.946526 (+ 752us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:48.947222 (+ 696us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:48.947512 (+ 290us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:48.949802 (+ 2290us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:48.949820 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:48.949831 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:08:48.949861 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:48.951673 (+ 1812us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:48.952091 (+ 418us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:48.952096 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:48.952100 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:48.952170 (+ 70us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:48.952395 (+ 225us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:48.952398 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:48.952400 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:48.952718 (+ 318us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:48.952910 (+ 192us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:48.953185 (+ 275us) server_negotiation.cc:300] Negotiation successful
0903 08:08:48.953478 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":178,"threads_started":1}
I20250903 08:08:48.954907 21347 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3aa107cc15d641bca7f21dd4251fdf15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aa107cc15d641bca7f21dd4251fdf15" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 38257 } }
I20250903 08:08:48.955264 21347 leader_election.cc:304] T 00000000000000000000000000000000 P 3aa107cc15d641bca7f21dd4251fdf15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3aa107cc15d641bca7f21dd4251fdf15; no voters:
I20250903 08:08:48.955570 21347 leader_election.cc:290] T 00000000000000000000000000000000 P 3aa107cc15d641bca7f21dd4251fdf15 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250903 08:08:48.955636 21352 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 3aa107cc15d641bca7f21dd4251fdf15 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 08:08:48.955821 21352 raft_consensus.cc:695] T 00000000000000000000000000000000 P 3aa107cc15d641bca7f21dd4251fdf15 [term 1 LEADER]: Becoming Leader. State: Replica: 3aa107cc15d641bca7f21dd4251fdf15, State: Running, Role: LEADER
I20250903 08:08:48.956102 21352 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3aa107cc15d641bca7f21dd4251fdf15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aa107cc15d641bca7f21dd4251fdf15" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 38257 } }
I20250903 08:08:48.956646 21347 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3aa107cc15d641bca7f21dd4251fdf15 [sys.catalog]: configured and running, proceeding with master startup.
I20250903 08:08:48.957176 21352 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3aa107cc15d641bca7f21dd4251fdf15 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3aa107cc15d641bca7f21dd4251fdf15. Latest consensus state: current_term: 1 leader_uuid: "3aa107cc15d641bca7f21dd4251fdf15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aa107cc15d641bca7f21dd4251fdf15" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 38257 } } }
I20250903 08:08:48.957286 21352 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3aa107cc15d641bca7f21dd4251fdf15 [sys.catalog]: This master's current role is: LEADER
I20250903 08:08:48.957614 21353 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3aa107cc15d641bca7f21dd4251fdf15 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3aa107cc15d641bca7f21dd4251fdf15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aa107cc15d641bca7f21dd4251fdf15" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 38257 } } }
I20250903 08:08:48.957758 21353 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3aa107cc15d641bca7f21dd4251fdf15 [sys.catalog]: This master's current role is: LEADER
I20250903 08:08:48.959013 21357 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250903 08:08:48.961526 21357 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250903 08:08:48.973263 21357 catalog_manager.cc:1349] Generated new cluster ID: 33fae318b477403190e5346926898f18
I20250903 08:08:48.973342 21357 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250903 08:08:48.985831 21357 catalog_manager.cc:1372] Generated new certificate authority record
I20250903 08:08:48.986847 21357 catalog_manager.cc:1506] Loading token signing keys...
I20250903 08:08:48.991539 21357 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 3aa107cc15d641bca7f21dd4251fdf15: Generated new TSK 0
I20250903 08:08:48.992220 21357 catalog_manager.cc:1516] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.12.201.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.201.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.201.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.12.201.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.12.201.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.201.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.201.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.12.201.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/krb5kdc/oryx.keytab.
I20250903 08:08:49.042495 13095 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUML90r/build/debug/bin/kudu
/tmp/dist-test-taskUML90r/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.201.193:0
--local_ip_for_outbound_sockets=127.12.201.193
--webserver_interface=127.12.201.193
--webserver_port=0
--tserver_master_addrs=127.12.201.254:38257
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/krb5kdc/oryx.keytab
--principal=oryx/127.12.201.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.201.212:42635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 08:08:49.150559 21375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:08:49.150804 21375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:08:49.150866 21375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:08:49.154233 21375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 08:08:49.154302 21375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:08:49.154356 21375 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20250903 08:08:49.154439 21375 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.201.193
I20250903 08:08:49.158839 21375 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.201.212:42635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/krb5kdc/oryx.keytab
--principal=oryx/127.12.201.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.201.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.201.193
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.201.254:38257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21375
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.201.193
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 07:50:08 UTC on 5fd53c4cbb9d
build id 7874
I20250903 08:08:49.159936 21375 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:08:49.160811 21375 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:08:49.167289 21383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:49.167300 21381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:49.167308 21380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:08:49.167408 21375 server_base.cc:1047] running on GCE node
I20250903 08:08:49.167896 21375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:08:49.168414 21375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:08:49.169554 21375 hybrid_clock.cc:648] HybridClock initialized: now 1756886929169531 us; error 38 us; skew 500 ppm
Sep 03 08:08:49 dist-test-slave-27r9 krb5kdc[21253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886929, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.201.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 08:08:49.172250 21375 init.cc:377] Logged in from keytab as oryx/127.12.201.193@KRBTEST.COM (short username oryx)
I20250903 08:08:49.173228 21375 webserver.cc:480] Webserver started at http://127.12.201.193:35991/ using document root <none> and password file <none>
I20250903 08:08:49.173748 21375 fs_manager.cc:362] Metadata directory not provided
I20250903 08:08:49.173799 21375 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:08:49.173985 21375 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:08:49.175734 21375 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/minicluster-data/ts-0/data/instance:
uuid: "fa5050fb31d1413a949c35c167d78324"
format_stamp: "Formatted at 2025-09-03 08:08:49 on dist-test-slave-27r9"
server_key: "94877c2aa12ba3482b26f9db34a8aca0"
server_key_iv: "f675d190030f9d8d1883860c297cf29c"
server_key_version: "encryptionkey@0"
I20250903 08:08:49.176203 21375 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/minicluster-data/ts-0/wal/instance:
uuid: "fa5050fb31d1413a949c35c167d78324"
format_stamp: "Formatted at 2025-09-03 08:08:49 on dist-test-slave-27r9"
server_key: "94877c2aa12ba3482b26f9db34a8aca0"
server_key_iv: "f675d190030f9d8d1883860c297cf29c"
server_key_version: "encryptionkey@0"
I20250903 08:08:49.179426 21375 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250903 08:08:49.181475 21390 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:49.182433 21375 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250903 08:08:49.182530 21375 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/minicluster-data/ts-0/wal
uuid: "fa5050fb31d1413a949c35c167d78324"
format_stamp: "Formatted at 2025-09-03 08:08:49 on dist-test-slave-27r9"
server_key: "94877c2aa12ba3482b26f9db34a8aca0"
server_key_iv: "f675d190030f9d8d1883860c297cf29c"
server_key_version: "encryptionkey@0"
I20250903 08:08:49.182621 21375 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:08:49.212958 21375 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:08:49.215672 21375 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:08:49.215847 21375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:08:49.217252 21375 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 08:08:49.217299 21375 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:49.217340 21375 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 08:08:49.217375 21375 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 03 08:08:49 dist-test-slave-27r9 krb5kdc[21253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886929, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.201.193@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM
I20250903 08:08:49.228314 21375 rpc_server.cc:307] RPC server started. Bound to: 127.12.201.193:34563
I20250903 08:08:49.228452 21505 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.201.193:34563 every 8 connection(s)
I20250903 08:08:49.229410 21375 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/minicluster-data/ts-0/data/info.pb
I20250903 08:08:49.229600 21350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:49.217969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.193:48495 (local address 127.12.201.254:38257)
0903 08:08:49.218113 (+ 144us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:49.218118 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:49.218900 (+ 782us) server_negotiation.cc:408] Connection header received
0903 08:08:49.220079 (+ 1179us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:49.220083 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:49.220141 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:49.220234 (+ 93us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:49.221644 (+ 1410us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:49.222200 (+ 556us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:49.222854 (+ 654us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:49.223083 (+ 229us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:49.225568 (+ 2485us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:49.225589 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:49.225591 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:08:49.225616 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:49.227560 (+ 1944us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:49.228024 (+ 464us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:49.228028 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:49.228030 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:49.228113 (+ 83us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:49.228456 (+ 343us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:49.228460 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:49.228462 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:49.228651 (+ 189us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:49.228733 (+ 82us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:49.229306 (+ 573us) server_negotiation.cc:300] Negotiation successful
0903 08:08:49.229420 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20250903 08:08:49.230279 21399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:49.218256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:38257 (local address 127.12.201.193:48495)
0903 08:08:49.218739 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:49.218787 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:49.219818 (+ 1031us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:49.220486 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:49.220501 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:49.220908 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:08:49.221487 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:49.221500 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:49.222322 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:49.222325 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:49.222747 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:49.222755 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:49.222943 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:49.223619 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:08:49.223655 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:08:49.225381 (+ 1726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:08:49.227676 (+ 2295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:49.227680 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:49.227688 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:49.227915 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:49.228233 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:49.228236 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:49.228239 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:49.228344 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:49.228756 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:08:49.228762 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:08:49.229033 (+ 271us) client_negotiation.cc:769] Sending connection context
0903 08:08:49.229258 (+ 225us) client_negotiation.cc:240] Negotiation successful
0903 08:08:49.229537 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":98,"threads_started":1}
I20250903 08:08:49.231392 21506 heartbeater.cc:344] Connected to a master server at 127.12.201.254:38257
I20250903 08:08:49.231541 21506 heartbeater.cc:461] Registering TS with master...
I20250903 08:08:49.232142 21506 heartbeater.cc:507] Master 127.12.201.254:38257 requested a full tablet report, sending...
I20250903 08:08:49.233510 21300 ts_manager.cc:194] Registered new tserver with Master: fa5050fb31d1413a949c35c167d78324 (127.12.201.193:34563)
I20250903 08:08:49.234645 21300 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.201.193@KRBTEST.COM'} at 127.12.201.193:48495
I20250903 08:08:49.238600 13095 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUML90r/build/debug/bin/kudu as pid 21375
I20250903 08:08:49.238713 13095 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/minicluster-data/ts-0/wal/instance
I20250903 08:08:49.239010 13095 external_mini_cluster.cc:1468] Setting key bead56008b018962010cd3f11e82868a
I20250903 08:08:49.239377 21350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:49.233479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.193:42829 (local address 127.12.201.254:38257)
0903 08:08:49.233679 (+ 200us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:49.233685 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:49.233699 (+ 14us) server_negotiation.cc:408] Connection header received
0903 08:08:49.233825 (+ 126us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:49.233828 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:49.233871 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:49.233935 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:49.234445 (+ 510us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:49.234947 (+ 502us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:49.235641 (+ 694us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:49.235772 (+ 131us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:49.236604 (+ 832us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:49.236617 (+ 13us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:49.236619 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:08:49.236657 (+ 38us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:49.237966 (+ 1309us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:49.238321 (+ 355us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:49.238324 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:49.238325 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:49.238359 (+ 34us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:49.238610 (+ 251us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:49.238612 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:49.238613 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:49.238775 (+ 162us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:49.238862 (+ 87us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:49.239177 (+ 315us) server_negotiation.cc:300] Negotiation successful
0903 08:08:49.239274 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20250903 08:08:49.239411 21512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:49.233410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:38257 (local address 127.12.201.193:42829)
0903 08:08:49.233639 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:49.233653 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:49.233741 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:49.233958 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:49.233961 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:49.234132 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:08:49.234333 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:49.234339 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:49.235070 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:49.235076 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:49.235495 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:49.235504 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:49.235650 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:49.236138 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:08:49.236153 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:08:49.236479 (+ 326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:08:49.238076 (+ 1597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:49.238079 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:49.238081 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:49.238237 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:49.238453 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:49.238456 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:49.238459 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:49.238527 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:49.238873 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:08:49.238877 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:08:49.238967 (+ 90us) client_negotiation.cc:769] Sending connection context
0903 08:08:49.239115 (+ 148us) client_negotiation.cc:240] Negotiation successful
0903 08:08:49.239242 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":80,"threads_started":1}
WARNING: no policy specified for oryx/127.12.201.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.201.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.201.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.12.201.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.12.201.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.201.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.201.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.12.201.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/krb5kdc/oryx.keytab.
I20250903 08:08:49.286134 13095 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUML90r/build/debug/bin/kudu
/tmp/dist-test-taskUML90r/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.201.194:0
--local_ip_for_outbound_sockets=127.12.201.194
--webserver_interface=127.12.201.194
--webserver_port=0
--tserver_master_addrs=127.12.201.254:38257
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/krb5kdc/oryx.keytab
--principal=oryx/127.12.201.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.201.212:42635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 08:08:49.389377 21517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:08:49.389601 21517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:08:49.389691 21517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:08:49.392903 21517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 08:08:49.392971 21517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:08:49.393062 21517 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20250903 08:08:49.393147 21517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.201.194
I20250903 08:08:49.397285 21517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.201.212:42635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/krb5kdc/oryx.keytab
--principal=oryx/127.12.201.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.201.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.201.194
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.201.254:38257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21517
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.201.194
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 07:50:08 UTC on 5fd53c4cbb9d
build id 7874
I20250903 08:08:49.398340 21517 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:08:49.399219 21517 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:08:49.405432 21522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:08:49.405546 21517 server_base.cc:1047] running on GCE node
W20250903 08:08:49.405485 21525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:49.405432 21523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:08:49.405985 21517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:08:49.406523 21517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:08:49.407675 21517 hybrid_clock.cc:648] HybridClock initialized: now 1756886929407657 us; error 33 us; skew 500 ppm
Sep 03 08:08:49 dist-test-slave-27r9 krb5kdc[21253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886929, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.201.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 08:08:49.410295 21517 init.cc:377] Logged in from keytab as oryx/127.12.201.194@KRBTEST.COM (short username oryx)
I20250903 08:08:49.411350 21517 webserver.cc:480] Webserver started at http://127.12.201.194:40937/ using document root <none> and password file <none>
I20250903 08:08:49.411875 21517 fs_manager.cc:362] Metadata directory not provided
I20250903 08:08:49.411928 21517 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:08:49.412102 21517 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:08:49.413836 21517 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/minicluster-data/ts-1/data/instance:
uuid: "9bf6ef1f80704b49bf1b4354718df875"
format_stamp: "Formatted at 2025-09-03 08:08:49 on dist-test-slave-27r9"
server_key: "a91b48d9b0d541c0ec986bac6d894b0b"
server_key_iv: "43a587ca55fe05934169976708aba03d"
server_key_version: "encryptionkey@0"
I20250903 08:08:49.414310 21517 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/minicluster-data/ts-1/wal/instance:
uuid: "9bf6ef1f80704b49bf1b4354718df875"
format_stamp: "Formatted at 2025-09-03 08:08:49 on dist-test-slave-27r9"
server_key: "a91b48d9b0d541c0ec986bac6d894b0b"
server_key_iv: "43a587ca55fe05934169976708aba03d"
server_key_version: "encryptionkey@0"
I20250903 08:08:49.417665 21517 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250903 08:08:49.419898 21532 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:49.420910 21517 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250903 08:08:49.421020 21517 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/minicluster-data/ts-1/wal
uuid: "9bf6ef1f80704b49bf1b4354718df875"
format_stamp: "Formatted at 2025-09-03 08:08:49 on dist-test-slave-27r9"
server_key: "a91b48d9b0d541c0ec986bac6d894b0b"
server_key_iv: "43a587ca55fe05934169976708aba03d"
server_key_version: "encryptionkey@0"
I20250903 08:08:49.421111 21517 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:08:49.433467 21517 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:08:49.436105 21517 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:08:49.436286 21517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:08:49.437640 21517 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 08:08:49.437700 21517 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:49.437755 21517 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 08:08:49.437783 21517 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 03 08:08:49 dist-test-slave-27r9 krb5kdc[21253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886929, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.201.194@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM
I20250903 08:08:49.449481 21517 rpc_server.cc:307] RPC server started. Bound to: 127.12.201.194:46253
I20250903 08:08:49.449520 21647 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.201.194:46253 every 8 connection(s)
I20250903 08:08:49.450601 21517 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/minicluster-data/ts-1/data/info.pb
I20250903 08:08:49.451175 21350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:49.438335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.194:56781 (local address 127.12.201.254:38257)
0903 08:08:49.438568 (+ 233us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:49.438573 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:49.439546 (+ 973us) server_negotiation.cc:408] Connection header received
0903 08:08:49.440814 (+ 1268us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:49.440819 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:49.440916 (+ 97us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:49.441029 (+ 113us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:49.442494 (+ 1465us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:49.443378 (+ 884us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:49.444276 (+ 898us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:49.444512 (+ 236us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:49.447216 (+ 2704us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:49.447245 (+ 29us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:49.447252 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:08:49.447286 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:49.449037 (+ 1751us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:49.449657 (+ 620us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:49.449661 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:49.449663 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:49.449712 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:49.449969 (+ 257us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:49.449972 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:49.449974 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:49.450152 (+ 178us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:49.450272 (+ 120us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:49.450705 (+ 433us) server_negotiation.cc:300] Negotiation successful
0903 08:08:49.450832 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20250903 08:08:49.451673 13095 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUML90r/build/debug/bin/kudu as pid 21517
I20250903 08:08:49.451786 13095 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/minicluster-data/ts-1/wal/instance
I20250903 08:08:49.452041 13095 external_mini_cluster.cc:1468] Setting key 833162f39aff6beac6b2418647a36121
I20250903 08:08:49.451618 21541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:49.438771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:38257 (local address 127.12.201.194:56781)
0903 08:08:49.439321 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:49.439401 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:49.440534 (+ 1133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:49.441166 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:49.441174 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:49.441573 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:08:49.442285 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:49.442307 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:49.443510 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:49.443514 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:49.444138 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:49.444146 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:49.444359 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:49.445089 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:08:49.445126 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:08:49.447057 (+ 1931us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:08:49.449166 (+ 2109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:49.449171 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:49.449183 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:49.449473 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:49.449814 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:49.449817 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:49.449818 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:49.449893 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:49.450258 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:08:49.450262 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:08:49.450474 (+ 212us) client_negotiation.cc:769] Sending connection context
0903 08:08:49.450631 (+ 157us) client_negotiation.cc:240] Negotiation successful
0903 08:08:49.450862 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":133,"threads_started":1}
I20250903 08:08:49.453701 21648 heartbeater.cc:344] Connected to a master server at 127.12.201.254:38257
I20250903 08:08:49.453933 21648 heartbeater.cc:461] Registering TS with master...
I20250903 08:08:49.454409 21648 heartbeater.cc:507] Master 127.12.201.254:38257 requested a full tablet report, sending...
I20250903 08:08:49.455264 21300 ts_manager.cc:194] Registered new tserver with Master: 9bf6ef1f80704b49bf1b4354718df875 (127.12.201.194:46253)
I20250903 08:08:49.455808 21300 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.201.194@KRBTEST.COM'} at 127.12.201.194:56781
WARNING: no policy specified for oryx/127.12.201.195@KRBTEST.COM; defaulting to no policy
I20250903 08:08:49.463336 21350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:49.457093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.194:48481 (local address 127.12.201.254:38257)
0903 08:08:49.457222 (+ 129us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:49.457226 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:49.457567 (+ 341us) server_negotiation.cc:408] Connection header received
0903 08:08:49.457683 (+ 116us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:49.457687 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:49.457731 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:49.457807 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:49.458358 (+ 551us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:49.458830 (+ 472us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:49.459482 (+ 652us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:49.459690 (+ 208us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:49.460495 (+ 805us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:49.460509 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:49.460511 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:08:49.460531 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:49.461926 (+ 1395us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:49.462284 (+ 358us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:49.462288 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:49.462289 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:49.462338 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:49.462562 (+ 224us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:49.462565 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:49.462567 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:49.462721 (+ 154us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:49.462810 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:49.463078 (+ 268us) server_negotiation.cc:300] Negotiation successful
0903 08:08:49.463189 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20250903 08:08:49.463368 21655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:49.457020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:38257 (local address 127.12.201.194:48481)
0903 08:08:49.457485 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:49.457499 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:49.457595 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:49.457824 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:49.457827 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:49.457996 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:08:49.458251 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:49.458258 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:49.458933 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:49.458936 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:49.459360 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:49.459368 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:49.459485 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:49.460097 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:08:49.460109 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:08:49.460368 (+ 259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:08:49.462024 (+ 1656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:49.462028 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:49.462030 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:49.462195 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:49.462437 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:49.462439 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:49.462441 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:49.462484 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:49.462809 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:08:49.462812 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:08:49.462888 (+ 76us) client_negotiation.cc:769] Sending connection context
0903 08:08:49.463037 (+ 149us) client_negotiation.cc:240] Negotiation successful
0903 08:08:49.463173 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":101,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.201.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.201.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.12.201.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.12.201.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.201.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.201.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.12.201.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/krb5kdc/oryx.keytab.
I20250903 08:08:49.497961 13095 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUML90r/build/debug/bin/kudu
/tmp/dist-test-taskUML90r/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.201.195:0
--local_ip_for_outbound_sockets=127.12.201.195
--webserver_interface=127.12.201.195
--webserver_port=0
--tserver_master_addrs=127.12.201.254:38257
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/krb5kdc/oryx.keytab
--principal=oryx/127.12.201.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.201.212:42635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 08:08:49.606029 21659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:08:49.606266 21659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:08:49.606333 21659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:08:49.609838 21659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 08:08:49.609915 21659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:08:49.609970 21659 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20250903 08:08:49.610044 21659 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.201.195
I20250903 08:08:49.614530 21659 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.201.212:42635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/krb5kdc/oryx.keytab
--principal=oryx/127.12.201.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.201.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.201.195
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.201.254:38257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21659
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.201.195
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 07:50:08 UTC on 5fd53c4cbb9d
build id 7874
I20250903 08:08:49.615634 21659 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:08:49.616487 21659 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:08:49.623265 21665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:49.623276 21667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:49.623299 21664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:08:49.623618 21659 server_base.cc:1047] running on GCE node
I20250903 08:08:49.623994 21659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:08:49.624572 21659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:08:49.625727 21659 hybrid_clock.cc:648] HybridClock initialized: now 1756886929625677 us; error 68 us; skew 500 ppm
Sep 03 08:08:49 dist-test-slave-27r9 krb5kdc[21253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886929, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.201.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 08:08:49.628610 21659 init.cc:377] Logged in from keytab as oryx/127.12.201.195@KRBTEST.COM (short username oryx)
I20250903 08:08:49.629698 21659 webserver.cc:480] Webserver started at http://127.12.201.195:41391/ using document root <none> and password file <none>
I20250903 08:08:49.630257 21659 fs_manager.cc:362] Metadata directory not provided
I20250903 08:08:49.630312 21659 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:08:49.630509 21659 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:08:49.632293 21659 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/minicluster-data/ts-2/data/instance:
uuid: "dc83179923a84bfaa56be4f55a7158ef"
format_stamp: "Formatted at 2025-09-03 08:08:49 on dist-test-slave-27r9"
server_key: "b32b5c4e89d53ba206e608fb67cdb95c"
server_key_iv: "21bd0cde724f08c16f386bd1fdd5a704"
server_key_version: "encryptionkey@0"
I20250903 08:08:49.632810 21659 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/minicluster-data/ts-2/wal/instance:
uuid: "dc83179923a84bfaa56be4f55a7158ef"
format_stamp: "Formatted at 2025-09-03 08:08:49 on dist-test-slave-27r9"
server_key: "b32b5c4e89d53ba206e608fb67cdb95c"
server_key_iv: "21bd0cde724f08c16f386bd1fdd5a704"
server_key_version: "encryptionkey@0"
I20250903 08:08:49.636240 21659 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20250903 08:08:49.638489 21674 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:49.639544 21659 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250903 08:08:49.639652 21659 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/minicluster-data/ts-2/wal
uuid: "dc83179923a84bfaa56be4f55a7158ef"
format_stamp: "Formatted at 2025-09-03 08:08:49 on dist-test-slave-27r9"
server_key: "b32b5c4e89d53ba206e608fb67cdb95c"
server_key_iv: "21bd0cde724f08c16f386bd1fdd5a704"
server_key_version: "encryptionkey@0"
I20250903 08:08:49.639750 21659 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:08:49.662580 21659 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:08:49.665557 21659 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:08:49.665748 21659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:08:49.667259 21659 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 08:08:49.667316 21659 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:49.667376 21659 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 08:08:49.667407 21659 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 03 08:08:49 dist-test-slave-27r9 krb5kdc[21253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886929, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.201.195@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM
I20250903 08:08:49.678849 21659 rpc_server.cc:307] RPC server started. Bound to: 127.12.201.195:36249
I20250903 08:08:49.678876 21789 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.201.195:36249 every 8 connection(s)
I20250903 08:08:49.679847 21659 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/minicluster-data/ts-2/data/info.pb
I20250903 08:08:49.680007 21350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:49.668008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.195:48365 (local address 127.12.201.254:38257)
0903 08:08:49.668152 (+ 144us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:49.668157 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:49.668956 (+ 799us) server_negotiation.cc:408] Connection header received
0903 08:08:49.670162 (+ 1206us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:49.670166 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:49.670231 (+ 65us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:49.670331 (+ 100us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:49.671948 (+ 1617us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:49.672444 (+ 496us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:49.673068 (+ 624us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:49.673323 (+ 255us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:49.676004 (+ 2681us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:49.676038 (+ 34us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:49.676043 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:08:49.676083 (+ 40us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:49.677970 (+ 1887us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:49.678466 (+ 496us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:49.678471 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:49.678473 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:49.678537 (+ 64us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:49.678894 (+ 357us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:49.678897 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:49.678899 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:49.679117 (+ 218us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:49.679216 (+ 99us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:49.679710 (+ 494us) server_negotiation.cc:300] Negotiation successful
0903 08:08:49.679835 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20250903 08:08:49.680675 21683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:49.668294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:38257 (local address 127.12.201.195:48365)
0903 08:08:49.668795 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:49.668845 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:49.669895 (+ 1050us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:49.670629 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:49.670638 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:49.671178 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:08:49.671778 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:49.671804 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:49.672560 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:49.672564 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:49.672953 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:49.672960 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:49.673162 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:49.674147 (+ 985us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:08:49.674185 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:08:49.675831 (+ 1646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:08:49.678108 (+ 2277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:49.678112 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:49.678122 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:49.678352 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:49.678658 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:49.678664 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:49.678669 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:49.678781 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:49.679227 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:08:49.679235 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:08:49.679468 (+ 233us) client_negotiation.cc:769] Sending connection context
0903 08:08:49.679656 (+ 188us) client_negotiation.cc:240] Negotiation successful
0903 08:08:49.679931 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":105,"threads_started":1}
I20250903 08:08:49.681993 21790 heartbeater.cc:344] Connected to a master server at 127.12.201.254:38257
I20250903 08:08:49.682183 21790 heartbeater.cc:461] Registering TS with master...
I20250903 08:08:49.682787 21790 heartbeater.cc:507] Master 127.12.201.254:38257 requested a full tablet report, sending...
I20250903 08:08:49.683917 13095 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUML90r/build/debug/bin/kudu as pid 21659
I20250903 08:08:49.683915 21301 ts_manager.cc:194] Registered new tserver with Master: dc83179923a84bfaa56be4f55a7158ef (127.12.201.195:36249)
I20250903 08:08:49.684041 13095 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/minicluster-data/ts-2/wal/instance
I20250903 08:08:49.684305 13095 external_mini_cluster.cc:1468] Setting key 99017664a3ff11882ccc22d14de79376
I20250903 08:08:49.684664 21301 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.201.195@KRBTEST.COM'} at 127.12.201.195:48365
I20250903 08:08:49.686239 13095 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250903 08:08:49.691203 21350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:49.684254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.195:55587 (local address 127.12.201.254:38257)
0903 08:08:49.684389 (+ 135us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:49.684395 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:49.684599 (+ 204us) server_negotiation.cc:408] Connection header received
0903 08:08:49.684721 (+ 122us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:49.684727 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:49.684772 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:49.684859 (+ 87us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:49.685518 (+ 659us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:49.686237 (+ 719us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:49.687059 (+ 822us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:49.687276 (+ 217us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:49.688197 (+ 921us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:49.688211 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:49.688214 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:08:49.688247 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:49.689743 (+ 1496us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:49.690138 (+ 395us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:49.690141 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:49.690142 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:49.690180 (+ 38us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:49.690468 (+ 288us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:49.690471 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:49.690473 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:49.690613 (+ 140us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:49.690701 (+ 88us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:49.690922 (+ 221us) server_negotiation.cc:300] Negotiation successful
0903 08:08:49.691046 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20250903 08:08:49.691200 21796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:49.684208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:38257 (local address 127.12.201.195:55587)
0903 08:08:49.684514 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:49.684526 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:49.684623 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:49.684895 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:49.684899 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:49.685111 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:08:49.685393 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:49.685402 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:49.686375 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:49.686378 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:49.686924 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:49.686932 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:49.687079 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:49.687730 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:08:49.687743 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:08:49.688062 (+ 319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:08:49.689863 (+ 1801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:49.689867 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:49.689869 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:49.690039 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:49.690314 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:49.690319 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:49.690322 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:49.690384 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:49.690698 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:08:49.690702 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:08:49.690777 (+ 75us) client_negotiation.cc:769] Sending connection context
0903 08:08:49.690865 (+ 88us) client_negotiation.cc:240] Negotiation successful
0903 08:08:49.691016 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":84,"threads_started":1}
Sep 03 08:08:49 dist-test-slave-27r9 krb5kdc[21253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886929, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 03 08:08:49 dist-test-slave-27r9 krb5kdc[21253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
Sep 03 08:08:49 dist-test-slave-27r9 krb5kdc[21253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
W20250903 08:08:49.708747 21350 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 08:08:49.703547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45658 (local address 127.12.201.254:38257)
0903 08:08:49.703677 (+ 130us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:49.703681 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:49.703695 (+ 14us) server_negotiation.cc:408] Connection header received
0903 08:08:49.703736 (+ 41us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:49.703739 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:49.703782 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:49.703858 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:49.704664 (+ 806us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:49.705137 (+ 473us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:49.705749 (+ 612us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:49.705868 (+ 119us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:49.708622 (+ 2754us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45658: BlockingRecv error: recv got EOF from 127.0.0.1:45658 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 03 08:08:49 dist-test-slave-27r9 krb5kdc[21253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886929, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM
I20250903 08:08:49.716868 21350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:49.710075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45672 (local address 127.12.201.254:38257)
0903 08:08:49.710211 (+ 136us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:49.710214 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:49.710240 (+ 26us) server_negotiation.cc:408] Connection header received
0903 08:08:49.710346 (+ 106us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:49.710348 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:49.710386 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:49.710453 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:49.711125 (+ 672us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:49.711611 (+ 486us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:49.712147 (+ 536us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:49.712283 (+ 136us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:49.714280 (+ 1997us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:49.714298 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:49.714300 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:08:49.714325 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:49.715720 (+ 1395us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:49.716035 (+ 315us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:49.716038 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:49.716039 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:49.716074 (+ 35us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:49.716233 (+ 159us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:49.716235 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:49.716236 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:49.716368 (+ 132us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:49.716466 (+ 98us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:49.716601 (+ 135us) server_negotiation.cc:300] Negotiation successful
0903 08:08:49.716687 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20250903 08:08:49.719550 21301 catalog_manager.cc:2232] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45672:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 08:08:49.721932 21301 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 08:08:49.734896 21812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:49.730144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.194:46253 (local address 127.0.0.1:49330)
0903 08:08:49.730463 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:49.730509 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:49.731099 (+ 590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:49.732116 (+ 1017us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:49.732119 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:49.732161 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 08:08:49.732498 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:49.732506 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:49.733646 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:49.733649 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:49.734516 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:49.734525 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:49.734596 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:49.734640 (+ 44us) client_negotiation.cc:769] Sending connection context
0903 08:08:49.734701 (+ 61us) client_negotiation.cc:240] Negotiation successful
0903 08:08:49.734776 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"spinlock_wait_cycles":11392,"thread_start_us":69,"threads_started":1}
I20250903 08:08:49.735433 21817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:49.730488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.195:36249 (local address 127.0.0.1:42216)
0903 08:08:49.731571 (+ 1083us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:49.731582 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:49.731659 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:49.732112 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:49.732118 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:49.732152 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 08:08:49.732500 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:49.732509 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:49.733993 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:49.733997 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:49.735026 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:49.735036 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:49.735167 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:49.735189 (+ 22us) client_negotiation.cc:769] Sending connection context
0903 08:08:49.735242 (+ 53us) client_negotiation.cc:240] Negotiation successful
0903 08:08:49.735300 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1018,"spinlock_wait_cycles":3968,"thread_start_us":117,"threads_started":1}
I20250903 08:08:49.736140 21814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:49.730751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49330 (local address 127.12.201.194:46253)
0903 08:08:49.731607 (+ 856us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:49.731611 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:49.731638 (+ 27us) server_negotiation.cc:408] Connection header received
0903 08:08:49.731728 (+ 90us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:49.731734 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:49.731932 (+ 198us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:49.732643 (+ 711us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 08:08:49.732796 (+ 153us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:49.733506 (+ 710us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:49.735196 (+ 1690us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:49.735762 (+ 566us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:49.735861 (+ 99us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:49.735917 (+ 56us) server_negotiation.cc:300] Negotiation successful
0903 08:08:49.735984 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":774,"thread_start_us":97,"threads_started":1}
I20250903 08:08:49.736223 21813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:49.730284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.193:34563 (local address 127.0.0.1:54102)
0903 08:08:49.731805 (+ 1521us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:49.731814 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:49.731885 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:49.732360 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:49.732364 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:49.732380 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 08:08:49.732582 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:49.732591 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:49.733993 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:49.733997 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:49.735027 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:49.735036 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:49.735933 (+ 897us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:49.735951 (+ 18us) client_negotiation.cc:769] Sending connection context
0903 08:08:49.736033 (+ 82us) client_negotiation.cc:240] Negotiation successful
0903 08:08:49.736092 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1476,"mutex_wait_us":944,"thread_start_us":93,"threads_started":1}
I20250903 08:08:49.736354 21816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:49.731073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54102 (local address 127.12.201.193:34563)
0903 08:08:49.731432 (+ 359us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:49.731440 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:49.731892 (+ 452us) server_negotiation.cc:408] Connection header received
0903 08:08:49.732032 (+ 140us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:49.732039 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:49.732222 (+ 183us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:49.732359 (+ 137us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 08:08:49.732711 (+ 352us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:49.733844 (+ 1133us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:49.735191 (+ 1347us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:49.735763 (+ 572us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:49.735869 (+ 106us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:49.736108 (+ 239us) server_negotiation.cc:300] Negotiation successful
0903 08:08:49.736217 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":90,"threads_started":1}
I20250903 08:08:49.736641 21815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:49.730919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42216 (local address 127.12.201.195:36249)
0903 08:08:49.731371 (+ 452us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:49.731378 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:49.731660 (+ 282us) server_negotiation.cc:408] Connection header received
0903 08:08:49.731752 (+ 92us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:49.731756 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:49.731959 (+ 203us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:49.732093 (+ 134us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 08:08:49.732637 (+ 544us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:49.733844 (+ 1207us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:49.735684 (+ 1840us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:49.736215 (+ 531us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:49.736321 (+ 106us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:49.736393 (+ 72us) server_negotiation.cc:300] Negotiation successful
0903 08:08:49.736497 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":137,"threads_started":1}
I20250903 08:08:49.738894 21724 tablet_service.cc:1468] Processing CreateTablet for tablet 321081ef9a2649069b101640791cee3c (DEFAULT_TABLE table=test-table [id=8c85b7eb0bb84ca58fc48728b152663f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:08:49.738896 21440 tablet_service.cc:1468] Processing CreateTablet for tablet 321081ef9a2649069b101640791cee3c (DEFAULT_TABLE table=test-table [id=8c85b7eb0bb84ca58fc48728b152663f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:08:49.739835 21440 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 321081ef9a2649069b101640791cee3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:08:49.739838 21724 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 321081ef9a2649069b101640791cee3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:08:49.741389 21582 tablet_service.cc:1468] Processing CreateTablet for tablet 321081ef9a2649069b101640791cee3c (DEFAULT_TABLE table=test-table [id=8c85b7eb0bb84ca58fc48728b152663f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:08:49.742372 21582 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 321081ef9a2649069b101640791cee3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:08:49.745474 21818 tablet_bootstrap.cc:492] T 321081ef9a2649069b101640791cee3c P fa5050fb31d1413a949c35c167d78324: Bootstrap starting.
I20250903 08:08:49.746816 21819 tablet_bootstrap.cc:492] T 321081ef9a2649069b101640791cee3c P dc83179923a84bfaa56be4f55a7158ef: Bootstrap starting.
I20250903 08:08:49.748205 21818 tablet_bootstrap.cc:654] T 321081ef9a2649069b101640791cee3c P fa5050fb31d1413a949c35c167d78324: Neither blocks nor log segments found. Creating new log.
I20250903 08:08:49.748670 21820 tablet_bootstrap.cc:492] T 321081ef9a2649069b101640791cee3c P 9bf6ef1f80704b49bf1b4354718df875: Bootstrap starting.
I20250903 08:08:49.748919 21819 tablet_bootstrap.cc:654] T 321081ef9a2649069b101640791cee3c P dc83179923a84bfaa56be4f55a7158ef: Neither blocks nor log segments found. Creating new log.
I20250903 08:08:49.749074 21818 log.cc:826] T 321081ef9a2649069b101640791cee3c P fa5050fb31d1413a949c35c167d78324: Log is configured to *not* fsync() on all Append() calls
I20250903 08:08:49.750056 21819 log.cc:826] T 321081ef9a2649069b101640791cee3c P dc83179923a84bfaa56be4f55a7158ef: Log is configured to *not* fsync() on all Append() calls
I20250903 08:08:49.750787 21820 tablet_bootstrap.cc:654] T 321081ef9a2649069b101640791cee3c P 9bf6ef1f80704b49bf1b4354718df875: Neither blocks nor log segments found. Creating new log.
I20250903 08:08:49.751008 21818 tablet_bootstrap.cc:492] T 321081ef9a2649069b101640791cee3c P fa5050fb31d1413a949c35c167d78324: No bootstrap required, opened a new log
I20250903 08:08:49.751243 21818 ts_tablet_manager.cc:1397] T 321081ef9a2649069b101640791cee3c P fa5050fb31d1413a949c35c167d78324: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20250903 08:08:49.751713 21819 tablet_bootstrap.cc:492] T 321081ef9a2649069b101640791cee3c P dc83179923a84bfaa56be4f55a7158ef: No bootstrap required, opened a new log
I20250903 08:08:49.751961 21819 ts_tablet_manager.cc:1397] T 321081ef9a2649069b101640791cee3c P dc83179923a84bfaa56be4f55a7158ef: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20250903 08:08:49.752004 21820 log.cc:826] T 321081ef9a2649069b101640791cee3c P 9bf6ef1f80704b49bf1b4354718df875: Log is configured to *not* fsync() on all Append() calls
I20250903 08:08:49.753832 21820 tablet_bootstrap.cc:492] T 321081ef9a2649069b101640791cee3c P 9bf6ef1f80704b49bf1b4354718df875: No bootstrap required, opened a new log
I20250903 08:08:49.754010 21820 ts_tablet_manager.cc:1397] T 321081ef9a2649069b101640791cee3c P 9bf6ef1f80704b49bf1b4354718df875: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250903 08:08:49.754712 21819 raft_consensus.cc:357] T 321081ef9a2649069b101640791cee3c P dc83179923a84bfaa56be4f55a7158ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bf6ef1f80704b49bf1b4354718df875" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 46253 } } peers { permanent_uuid: "fa5050fb31d1413a949c35c167d78324" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 34563 } } peers { permanent_uuid: "dc83179923a84bfaa56be4f55a7158ef" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 36249 } }
I20250903 08:08:49.754695 21818 raft_consensus.cc:357] T 321081ef9a2649069b101640791cee3c P fa5050fb31d1413a949c35c167d78324 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bf6ef1f80704b49bf1b4354718df875" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 46253 } } peers { permanent_uuid: "fa5050fb31d1413a949c35c167d78324" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 34563 } } peers { permanent_uuid: "dc83179923a84bfaa56be4f55a7158ef" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 36249 } }
I20250903 08:08:49.754897 21818 raft_consensus.cc:383] T 321081ef9a2649069b101640791cee3c P fa5050fb31d1413a949c35c167d78324 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:08:49.754896 21819 raft_consensus.cc:383] T 321081ef9a2649069b101640791cee3c P dc83179923a84bfaa56be4f55a7158ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:08:49.754961 21819 raft_consensus.cc:738] T 321081ef9a2649069b101640791cee3c P dc83179923a84bfaa56be4f55a7158ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc83179923a84bfaa56be4f55a7158ef, State: Initialized, Role: FOLLOWER
I20250903 08:08:49.755017 21818 raft_consensus.cc:738] T 321081ef9a2649069b101640791cee3c P fa5050fb31d1413a949c35c167d78324 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa5050fb31d1413a949c35c167d78324, State: Initialized, Role: FOLLOWER
I20250903 08:08:49.755430 21819 consensus_queue.cc:260] T 321081ef9a2649069b101640791cee3c P dc83179923a84bfaa56be4f55a7158ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bf6ef1f80704b49bf1b4354718df875" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 46253 } } peers { permanent_uuid: "fa5050fb31d1413a949c35c167d78324" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 34563 } } peers { permanent_uuid: "dc83179923a84bfaa56be4f55a7158ef" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 36249 } }
I20250903 08:08:49.755517 21818 consensus_queue.cc:260] T 321081ef9a2649069b101640791cee3c P fa5050fb31d1413a949c35c167d78324 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bf6ef1f80704b49bf1b4354718df875" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 46253 } } peers { permanent_uuid: "fa5050fb31d1413a949c35c167d78324" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 34563 } } peers { permanent_uuid: "dc83179923a84bfaa56be4f55a7158ef" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 36249 } }
I20250903 08:08:49.756088 21790 heartbeater.cc:499] Master 127.12.201.254:38257 was elected leader, sending a full tablet report...
I20250903 08:08:49.756352 21819 ts_tablet_manager.cc:1428] T 321081ef9a2649069b101640791cee3c P dc83179923a84bfaa56be4f55a7158ef: Time spent starting tablet: real 0.004s user 0.001s sys 0.003s
I20250903 08:08:49.756502 21818 ts_tablet_manager.cc:1428] T 321081ef9a2649069b101640791cee3c P fa5050fb31d1413a949c35c167d78324: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
I20250903 08:08:49.756876 21506 heartbeater.cc:499] Master 127.12.201.254:38257 was elected leader, sending a full tablet report...
I20250903 08:08:49.757068 21820 raft_consensus.cc:357] T 321081ef9a2649069b101640791cee3c P 9bf6ef1f80704b49bf1b4354718df875 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bf6ef1f80704b49bf1b4354718df875" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 46253 } } peers { permanent_uuid: "fa5050fb31d1413a949c35c167d78324" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 34563 } } peers { permanent_uuid: "dc83179923a84bfaa56be4f55a7158ef" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 36249 } }
I20250903 08:08:49.757304 21820 raft_consensus.cc:383] T 321081ef9a2649069b101640791cee3c P 9bf6ef1f80704b49bf1b4354718df875 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:08:49.757376 21820 raft_consensus.cc:738] T 321081ef9a2649069b101640791cee3c P 9bf6ef1f80704b49bf1b4354718df875 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bf6ef1f80704b49bf1b4354718df875, State: Initialized, Role: FOLLOWER
I20250903 08:08:49.757782 21820 consensus_queue.cc:260] T 321081ef9a2649069b101640791cee3c P 9bf6ef1f80704b49bf1b4354718df875 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bf6ef1f80704b49bf1b4354718df875" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 46253 } } peers { permanent_uuid: "fa5050fb31d1413a949c35c167d78324" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 34563 } } peers { permanent_uuid: "dc83179923a84bfaa56be4f55a7158ef" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 36249 } }
I20250903 08:08:49.758597 21820 ts_tablet_manager.cc:1428] T 321081ef9a2649069b101640791cee3c P 9bf6ef1f80704b49bf1b4354718df875: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20250903 08:08:49.758852 21648 heartbeater.cc:499] Master 127.12.201.254:38257 was elected leader, sending a full tablet report...
I20250903 08:08:49.820839 21825 raft_consensus.cc:491] T 321081ef9a2649069b101640791cee3c P fa5050fb31d1413a949c35c167d78324 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 08:08:49.821045 21825 raft_consensus.cc:513] T 321081ef9a2649069b101640791cee3c P fa5050fb31d1413a949c35c167d78324 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bf6ef1f80704b49bf1b4354718df875" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 46253 } } peers { permanent_uuid: "fa5050fb31d1413a949c35c167d78324" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 34563 } } peers { permanent_uuid: "dc83179923a84bfaa56be4f55a7158ef" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 36249 } }
I20250903 08:08:49.822225 21825 leader_election.cc:290] T 321081ef9a2649069b101640791cee3c P fa5050fb31d1413a949c35c167d78324 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bf6ef1f80704b49bf1b4354718df875 (127.12.201.194:46253), dc83179923a84bfaa56be4f55a7158ef (127.12.201.195:36249)
I20250903 08:08:49.825407 21827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:49.822432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.194:46253 (local address 127.12.201.193:51963)
0903 08:08:49.822866 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:49.822879 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:49.823025 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:49.823304 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:49.823307 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:49.823359 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 08:08:49.823615 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:49.823621 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:49.824369 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:49.824372 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:49.825005 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:49.825013 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:49.825156 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:49.825177 (+ 21us) client_negotiation.cc:769] Sending connection context
0903 08:08:49.825218 (+ 41us) client_negotiation.cc:240] Negotiation successful
0903 08:08:49.825275 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"spinlock_wait_cycles":5888,"thread_start_us":121,"threads_started":1}
I20250903 08:08:49.825944 21814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:49.822616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.193:51963 (local address 127.12.201.194:46253)
0903 08:08:49.822754 (+ 138us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:49.822759 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:49.822969 (+ 210us) server_negotiation.cc:408] Connection header received
0903 08:08:49.823140 (+ 171us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:49.823143 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:49.823210 (+ 67us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:49.823280 (+ 70us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 08:08:49.823735 (+ 455us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:49.824241 (+ 506us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:49.825145 (+ 904us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:49.825676 (+ 531us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:49.825718 (+ 42us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:49.825767 (+ 49us) server_negotiation.cc:300] Negotiation successful
0903 08:08:49.825828 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20250903 08:08:49.826548 21815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:49.822615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.193:37893 (local address 127.12.201.195:36249)
0903 08:08:49.822754 (+ 139us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:49.822759 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:49.823007 (+ 248us) server_negotiation.cc:408] Connection header received
0903 08:08:49.823166 (+ 159us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:49.823169 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:49.823218 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:49.823287 (+ 69us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 08:08:49.824118 (+ 831us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:49.824603 (+ 485us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:49.825649 (+ 1046us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:49.826111 (+ 462us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:49.826145 (+ 34us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:49.826351 (+ 206us) server_negotiation.cc:300] Negotiation successful
0903 08:08:49.826424 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20250903 08:08:49.826586 21602 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "321081ef9a2649069b101640791cee3c" candidate_uuid: "fa5050fb31d1413a949c35c167d78324" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bf6ef1f80704b49bf1b4354718df875" is_pre_election: true
I20250903 08:08:49.826845 21828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:49.822432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.195:36249 (local address 127.12.201.193:37893)
0903 08:08:49.822912 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:49.822924 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:49.823066 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:49.823303 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:49.823306 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:49.823354 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 08:08:49.823992 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:49.824000 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:49.824708 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:49.824732 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:49.825498 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:49.825508 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:49.826206 (+ 698us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:49.826222 (+ 16us) client_negotiation.cc:769] Sending connection context
0903 08:08:49.826670 (+ 448us) client_negotiation.cc:240] Negotiation successful
0903 08:08:49.826739 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"mutex_wait_us":115,"spinlock_wait_cycles":27264,"thread_start_us":127,"threads_started":1}
I20250903 08:08:49.826890 21602 raft_consensus.cc:2466] T 321081ef9a2649069b101640791cee3c P 9bf6ef1f80704b49bf1b4354718df875 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa5050fb31d1413a949c35c167d78324 in term 0.
I20250903 08:08:49.827365 21392 leader_election.cc:304] T 321081ef9a2649069b101640791cee3c P fa5050fb31d1413a949c35c167d78324 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bf6ef1f80704b49bf1b4354718df875, fa5050fb31d1413a949c35c167d78324; no voters:
I20250903 08:08:49.827437 21744 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "321081ef9a2649069b101640791cee3c" candidate_uuid: "fa5050fb31d1413a949c35c167d78324" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc83179923a84bfaa56be4f55a7158ef" is_pre_election: true
I20250903 08:08:49.827626 21825 raft_consensus.cc:2802] T 321081ef9a2649069b101640791cee3c P fa5050fb31d1413a949c35c167d78324 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 08:08:49.827710 21825 raft_consensus.cc:491] T 321081ef9a2649069b101640791cee3c P fa5050fb31d1413a949c35c167d78324 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 08:08:49.827694 21744 raft_consensus.cc:2466] T 321081ef9a2649069b101640791cee3c P dc83179923a84bfaa56be4f55a7158ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa5050fb31d1413a949c35c167d78324 in term 0.
I20250903 08:08:49.827759 21825 raft_consensus.cc:3058] T 321081ef9a2649069b101640791cee3c P fa5050fb31d1413a949c35c167d78324 [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:08:49.828692 21825 raft_consensus.cc:513] T 321081ef9a2649069b101640791cee3c P fa5050fb31d1413a949c35c167d78324 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bf6ef1f80704b49bf1b4354718df875" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 46253 } } peers { permanent_uuid: "fa5050fb31d1413a949c35c167d78324" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 34563 } } peers { permanent_uuid: "dc83179923a84bfaa56be4f55a7158ef" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 36249 } }
I20250903 08:08:49.829001 21825 leader_election.cc:290] T 321081ef9a2649069b101640791cee3c P fa5050fb31d1413a949c35c167d78324 [CANDIDATE]: Term 1 election: Requested vote from peers 9bf6ef1f80704b49bf1b4354718df875 (127.12.201.194:46253), dc83179923a84bfaa56be4f55a7158ef (127.12.201.195:36249)
I20250903 08:08:49.829324 21744 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "321081ef9a2649069b101640791cee3c" candidate_uuid: "fa5050fb31d1413a949c35c167d78324" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc83179923a84bfaa56be4f55a7158ef"
I20250903 08:08:49.829324 21602 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "321081ef9a2649069b101640791cee3c" candidate_uuid: "fa5050fb31d1413a949c35c167d78324" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bf6ef1f80704b49bf1b4354718df875"
I20250903 08:08:49.829434 21602 raft_consensus.cc:3058] T 321081ef9a2649069b101640791cee3c P 9bf6ef1f80704b49bf1b4354718df875 [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:08:49.829459 21744 raft_consensus.cc:3058] T 321081ef9a2649069b101640791cee3c P dc83179923a84bfaa56be4f55a7158ef [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:08:49.830649 21602 raft_consensus.cc:2466] T 321081ef9a2649069b101640791cee3c P 9bf6ef1f80704b49bf1b4354718df875 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa5050fb31d1413a949c35c167d78324 in term 1.
I20250903 08:08:49.830649 21744 raft_consensus.cc:2466] T 321081ef9a2649069b101640791cee3c P dc83179923a84bfaa56be4f55a7158ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa5050fb31d1413a949c35c167d78324 in term 1.
I20250903 08:08:49.830919 21392 leader_election.cc:304] T 321081ef9a2649069b101640791cee3c P fa5050fb31d1413a949c35c167d78324 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bf6ef1f80704b49bf1b4354718df875, fa5050fb31d1413a949c35c167d78324; no voters:
I20250903 08:08:49.831087 21825 raft_consensus.cc:2802] T 321081ef9a2649069b101640791cee3c P fa5050fb31d1413a949c35c167d78324 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 08:08:49.831362 21825 raft_consensus.cc:695] T 321081ef9a2649069b101640791cee3c P fa5050fb31d1413a949c35c167d78324 [term 1 LEADER]: Becoming Leader. State: Replica: fa5050fb31d1413a949c35c167d78324, State: Running, Role: LEADER
I20250903 08:08:49.831651 21825 consensus_queue.cc:237] T 321081ef9a2649069b101640791cee3c P fa5050fb31d1413a949c35c167d78324 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bf6ef1f80704b49bf1b4354718df875" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 46253 } } peers { permanent_uuid: "fa5050fb31d1413a949c35c167d78324" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 34563 } } peers { permanent_uuid: "dc83179923a84bfaa56be4f55a7158ef" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 36249 } }
I20250903 08:08:49.834698 21300 catalog_manager.cc:5582] T 321081ef9a2649069b101640791cee3c P fa5050fb31d1413a949c35c167d78324 reported cstate change: term changed from 0 to 1, leader changed from <none> to fa5050fb31d1413a949c35c167d78324 (127.12.201.193). New cstate: current_term: 1 leader_uuid: "fa5050fb31d1413a949c35c167d78324" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bf6ef1f80704b49bf1b4354718df875" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 46253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa5050fb31d1413a949c35c167d78324" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 34563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc83179923a84bfaa56be4f55a7158ef" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 36249 } health_report { overall_health: UNKNOWN } } }
I20250903 08:08:49.857206 21816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:49.854378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54108 (local address 127.12.201.193:34563)
0903 08:08:49.854525 (+ 147us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:49.854529 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:49.854542 (+ 13us) server_negotiation.cc:408] Connection header received
0903 08:08:49.854618 (+ 76us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:49.854620 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:49.854674 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:49.854757 (+ 83us) server_negotiation.cc:227] Negotiated authn=TOKEN
0903 08:08:49.855100 (+ 343us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:49.855640 (+ 540us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:49.856259 (+ 619us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:49.856395 (+ 136us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:49.856444 (+ 49us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0903 08:08:49.856906 (+ 462us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0903 08:08:49.856998 (+ 92us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:49.857029 (+ 31us) server_negotiation.cc:300] Negotiation successful
0903 08:08:49.857072 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20250903 08:08:49.862701 21744 raft_consensus.cc:1273] T 321081ef9a2649069b101640791cee3c P dc83179923a84bfaa56be4f55a7158ef [term 1 FOLLOWER]: Refusing update from remote peer fa5050fb31d1413a949c35c167d78324: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 08:08:49.862701 21602 raft_consensus.cc:1273] T 321081ef9a2649069b101640791cee3c P 9bf6ef1f80704b49bf1b4354718df875 [term 1 FOLLOWER]: Refusing update from remote peer fa5050fb31d1413a949c35c167d78324: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 08:08:49.863405 21825 consensus_queue.cc:1035] T 321081ef9a2649069b101640791cee3c P fa5050fb31d1413a949c35c167d78324 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc83179923a84bfaa56be4f55a7158ef" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 36249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 08:08:49.863634 21829 consensus_queue.cc:1035] T 321081ef9a2649069b101640791cee3c P fa5050fb31d1413a949c35c167d78324 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bf6ef1f80704b49bf1b4354718df875" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 46253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 08:08:49.870762 21832 mvcc.cc:204] Tried to move back new op lower bound from 7196208864710746112 to 7196208864593281024. Current Snapshot: MvccSnapshot[applied={T|T < 7196208864710746112}]
I20250903 08:08:49.873669 21833 mvcc.cc:204] Tried to move back new op lower bound from 7196208864710746112 to 7196208864593281024. Current Snapshot: MvccSnapshot[applied={T|T < 7196208864710746112}]
I20250903 08:08:49.880376 21300 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45672:
table { table_name: "test-table" } modify_external_catalogs: true
I20250903 08:08:49.880539 21300 catalog_manager.cc:2730] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45672:
table { table_name: "test-table" } modify_external_catalogs: true
I20250903 08:08:49.883105 21300 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3aa107cc15d641bca7f21dd4251fdf15: Sending DeleteTablet for 3 replicas of tablet 321081ef9a2649069b101640791cee3c
I20250903 08:08:49.883847 21582 tablet_service.cc:1515] Processing DeleteTablet for tablet 321081ef9a2649069b101640791cee3c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 08:08:49 UTC) from {username='oryx', principal='oryx/127.12.201.254@KRBTEST.COM'} at 127.0.0.1:49330
I20250903 08:08:49.883903 21440 tablet_service.cc:1515] Processing DeleteTablet for tablet 321081ef9a2649069b101640791cee3c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 08:08:49 UTC) from {username='oryx', principal='oryx/127.12.201.254@KRBTEST.COM'} at 127.0.0.1:54102
I20250903 08:08:49.884371 21724 tablet_service.cc:1515] Processing DeleteTablet for tablet 321081ef9a2649069b101640791cee3c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 08:08:49 UTC) from {username='oryx', principal='oryx/127.12.201.254@KRBTEST.COM'} at 127.0.0.1:42216
I20250903 08:08:49.884698 21842 tablet_replica.cc:331] T 321081ef9a2649069b101640791cee3c P fa5050fb31d1413a949c35c167d78324: stopping tablet replica
I20250903 08:08:49.884822 21843 tablet_replica.cc:331] T 321081ef9a2649069b101640791cee3c P dc83179923a84bfaa56be4f55a7158ef: stopping tablet replica
I20250903 08:08:49.884825 21300 catalog_manager.cc:2232] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45672:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 08:08:49.884917 21841 tablet_replica.cc:331] T 321081ef9a2649069b101640791cee3c P 9bf6ef1f80704b49bf1b4354718df875: stopping tablet replica
I20250903 08:08:49.885133 21843 raft_consensus.cc:2241] T 321081ef9a2649069b101640791cee3c P dc83179923a84bfaa56be4f55a7158ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 08:08:49.885143 21841 raft_consensus.cc:2241] T 321081ef9a2649069b101640791cee3c P 9bf6ef1f80704b49bf1b4354718df875 [term 1 FOLLOWER]: Raft consensus shutting down.
W20250903 08:08:49.885262 21300 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 08:08:49.885460 21843 raft_consensus.cc:2270] T 321081ef9a2649069b101640791cee3c P dc83179923a84bfaa56be4f55a7158ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 08:08:49.885466 21841 raft_consensus.cc:2270] T 321081ef9a2649069b101640791cee3c P 9bf6ef1f80704b49bf1b4354718df875 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 08:08:49.885866 21842 raft_consensus.cc:2241] T 321081ef9a2649069b101640791cee3c P fa5050fb31d1413a949c35c167d78324 [term 1 LEADER]: Raft consensus shutting down.
I20250903 08:08:49.886289 21842 raft_consensus.cc:2270] T 321081ef9a2649069b101640791cee3c P fa5050fb31d1413a949c35c167d78324 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 08:08:49.887378 21843 ts_tablet_manager.cc:1905] T 321081ef9a2649069b101640791cee3c P dc83179923a84bfaa56be4f55a7158ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 08:08:49.887526 21842 ts_tablet_manager.cc:1905] T 321081ef9a2649069b101640791cee3c P fa5050fb31d1413a949c35c167d78324: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 08:08:49.888276 21841 ts_tablet_manager.cc:1905] T 321081ef9a2649069b101640791cee3c P 9bf6ef1f80704b49bf1b4354718df875: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 08:08:49.890285 21842 ts_tablet_manager.cc:1918] T 321081ef9a2649069b101640791cee3c P fa5050fb31d1413a949c35c167d78324: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250903 08:08:49.890434 21842 log.cc:1199] T 321081ef9a2649069b101640791cee3c P fa5050fb31d1413a949c35c167d78324: Deleting WAL directory at /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/minicluster-data/ts-0/wal/wals/321081ef9a2649069b101640791cee3c
I20250903 08:08:49.890813 21843 ts_tablet_manager.cc:1918] T 321081ef9a2649069b101640791cee3c P dc83179923a84bfaa56be4f55a7158ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250903 08:08:49.890872 21842 ts_tablet_manager.cc:1939] T 321081ef9a2649069b101640791cee3c P fa5050fb31d1413a949c35c167d78324: Deleting consensus metadata
I20250903 08:08:49.892023 21288 catalog_manager.cc:4928] TS fa5050fb31d1413a949c35c167d78324 (127.12.201.193:34563): tablet 321081ef9a2649069b101640791cee3c (table test-table [id=8c85b7eb0bb84ca58fc48728b152663f]) successfully deleted
I20250903 08:08:49.892138 21440 tablet_service.cc:1468] Processing CreateTablet for tablet 6b660cab5f9e41768b2074130cbe8a4d (DEFAULT_TABLE table=test-table [id=f552d51104834d7e9b3a775fb0468492]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:08:49.892391 21440 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b660cab5f9e41768b2074130cbe8a4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:08:49.893333 21841 ts_tablet_manager.cc:1918] T 321081ef9a2649069b101640791cee3c P 9bf6ef1f80704b49bf1b4354718df875: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250903 08:08:49.893416 21841 log.cc:1199] T 321081ef9a2649069b101640791cee3c P 9bf6ef1f80704b49bf1b4354718df875: Deleting WAL directory at /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/minicluster-data/ts-1/wal/wals/321081ef9a2649069b101640791cee3c
I20250903 08:08:49.893746 21841 ts_tablet_manager.cc:1939] T 321081ef9a2649069b101640791cee3c P 9bf6ef1f80704b49bf1b4354718df875: Deleting consensus metadata
I20250903 08:08:49.893965 21818 tablet_bootstrap.cc:492] T 6b660cab5f9e41768b2074130cbe8a4d P fa5050fb31d1413a949c35c167d78324: Bootstrap starting.
I20250903 08:08:49.894901 21286 catalog_manager.cc:4928] TS 9bf6ef1f80704b49bf1b4354718df875 (127.12.201.194:46253): tablet 321081ef9a2649069b101640791cee3c (table test-table [id=8c85b7eb0bb84ca58fc48728b152663f]) successfully deleted
I20250903 08:08:49.895089 21818 tablet_bootstrap.cc:654] T 6b660cab5f9e41768b2074130cbe8a4d P fa5050fb31d1413a949c35c167d78324: Neither blocks nor log segments found. Creating new log.
I20250903 08:08:49.894181 21724 tablet_service.cc:1468] Processing CreateTablet for tablet 6b660cab5f9e41768b2074130cbe8a4d (DEFAULT_TABLE table=test-table [id=f552d51104834d7e9b3a775fb0468492]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:08:49.895171 21582 tablet_service.cc:1468] Processing CreateTablet for tablet 6b660cab5f9e41768b2074130cbe8a4d (DEFAULT_TABLE table=test-table [id=f552d51104834d7e9b3a775fb0468492]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:08:49.895341 21724 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b660cab5f9e41768b2074130cbe8a4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:08:49.895431 21582 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b660cab5f9e41768b2074130cbe8a4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:08:49.896354 21818 tablet_bootstrap.cc:492] T 6b660cab5f9e41768b2074130cbe8a4d P fa5050fb31d1413a949c35c167d78324: No bootstrap required, opened a new log
I20250903 08:08:49.896438 21818 ts_tablet_manager.cc:1397] T 6b660cab5f9e41768b2074130cbe8a4d P fa5050fb31d1413a949c35c167d78324: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20250903 08:08:49.896961 21818 raft_consensus.cc:357] T 6b660cab5f9e41768b2074130cbe8a4d P fa5050fb31d1413a949c35c167d78324 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5050fb31d1413a949c35c167d78324" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 34563 } } peers { permanent_uuid: "9bf6ef1f80704b49bf1b4354718df875" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 46253 } } peers { permanent_uuid: "dc83179923a84bfaa56be4f55a7158ef" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 36249 } }
I20250903 08:08:49.897064 21818 raft_consensus.cc:383] T 6b660cab5f9e41768b2074130cbe8a4d P fa5050fb31d1413a949c35c167d78324 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:08:49.897094 21818 raft_consensus.cc:738] T 6b660cab5f9e41768b2074130cbe8a4d P fa5050fb31d1413a949c35c167d78324 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa5050fb31d1413a949c35c167d78324, State: Initialized, Role: FOLLOWER
I20250903 08:08:49.897219 21818 consensus_queue.cc:260] T 6b660cab5f9e41768b2074130cbe8a4d P fa5050fb31d1413a949c35c167d78324 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5050fb31d1413a949c35c167d78324" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 34563 } } peers { permanent_uuid: "9bf6ef1f80704b49bf1b4354718df875" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 46253 } } peers { permanent_uuid: "dc83179923a84bfaa56be4f55a7158ef" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 36249 } }
I20250903 08:08:49.897527 21818 ts_tablet_manager.cc:1428] T 6b660cab5f9e41768b2074130cbe8a4d P fa5050fb31d1413a949c35c167d78324: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250903 08:08:49.897559 21819 tablet_bootstrap.cc:492] T 6b660cab5f9e41768b2074130cbe8a4d P dc83179923a84bfaa56be4f55a7158ef: Bootstrap starting.
I20250903 08:08:49.897848 21820 tablet_bootstrap.cc:492] T 6b660cab5f9e41768b2074130cbe8a4d P 9bf6ef1f80704b49bf1b4354718df875: Bootstrap starting.
I20250903 08:08:49.898562 21819 tablet_bootstrap.cc:654] T 6b660cab5f9e41768b2074130cbe8a4d P dc83179923a84bfaa56be4f55a7158ef: Neither blocks nor log segments found. Creating new log.
I20250903 08:08:49.898840 21820 tablet_bootstrap.cc:654] T 6b660cab5f9e41768b2074130cbe8a4d P 9bf6ef1f80704b49bf1b4354718df875: Neither blocks nor log segments found. Creating new log.
I20250903 08:08:49.899717 21843 log.cc:1199] T 321081ef9a2649069b101640791cee3c P dc83179923a84bfaa56be4f55a7158ef: Deleting WAL directory at /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/minicluster-data/ts-2/wal/wals/321081ef9a2649069b101640791cee3c
I20250903 08:08:49.900527 21819 tablet_bootstrap.cc:492] T 6b660cab5f9e41768b2074130cbe8a4d P dc83179923a84bfaa56be4f55a7158ef: No bootstrap required, opened a new log
I20250903 08:08:49.900611 21819 ts_tablet_manager.cc:1397] T 6b660cab5f9e41768b2074130cbe8a4d P dc83179923a84bfaa56be4f55a7158ef: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250903 08:08:49.901036 21819 raft_consensus.cc:357] T 6b660cab5f9e41768b2074130cbe8a4d P dc83179923a84bfaa56be4f55a7158ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5050fb31d1413a949c35c167d78324" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 34563 } } peers { permanent_uuid: "9bf6ef1f80704b49bf1b4354718df875" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 46253 } } peers { permanent_uuid: "dc83179923a84bfaa56be4f55a7158ef" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 36249 } }
I20250903 08:08:49.901131 21819 raft_consensus.cc:383] T 6b660cab5f9e41768b2074130cbe8a4d P dc83179923a84bfaa56be4f55a7158ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:08:49.901161 21819 raft_consensus.cc:738] T 6b660cab5f9e41768b2074130cbe8a4d P dc83179923a84bfaa56be4f55a7158ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc83179923a84bfaa56be4f55a7158ef, State: Initialized, Role: FOLLOWER
I20250903 08:08:49.901280 21819 consensus_queue.cc:260] T 6b660cab5f9e41768b2074130cbe8a4d P dc83179923a84bfaa56be4f55a7158ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5050fb31d1413a949c35c167d78324" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 34563 } } peers { permanent_uuid: "9bf6ef1f80704b49bf1b4354718df875" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 46253 } } peers { permanent_uuid: "dc83179923a84bfaa56be4f55a7158ef" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 36249 } }
I20250903 08:08:49.901520 21819 ts_tablet_manager.cc:1428] T 6b660cab5f9e41768b2074130cbe8a4d P dc83179923a84bfaa56be4f55a7158ef: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250903 08:08:49.901765 21843 ts_tablet_manager.cc:1939] T 321081ef9a2649069b101640791cee3c P dc83179923a84bfaa56be4f55a7158ef: Deleting consensus metadata
I20250903 08:08:49.901819 21820 tablet_bootstrap.cc:492] T 6b660cab5f9e41768b2074130cbe8a4d P 9bf6ef1f80704b49bf1b4354718df875: No bootstrap required, opened a new log
I20250903 08:08:49.901919 21820 ts_tablet_manager.cc:1397] T 6b660cab5f9e41768b2074130cbe8a4d P 9bf6ef1f80704b49bf1b4354718df875: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20250903 08:08:49.902441 21820 raft_consensus.cc:357] T 6b660cab5f9e41768b2074130cbe8a4d P 9bf6ef1f80704b49bf1b4354718df875 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5050fb31d1413a949c35c167d78324" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 34563 } } peers { permanent_uuid: "9bf6ef1f80704b49bf1b4354718df875" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 46253 } } peers { permanent_uuid: "dc83179923a84bfaa56be4f55a7158ef" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 36249 } }
I20250903 08:08:49.902571 21820 raft_consensus.cc:383] T 6b660cab5f9e41768b2074130cbe8a4d P 9bf6ef1f80704b49bf1b4354718df875 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:08:49.902616 21820 raft_consensus.cc:738] T 6b660cab5f9e41768b2074130cbe8a4d P 9bf6ef1f80704b49bf1b4354718df875 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bf6ef1f80704b49bf1b4354718df875, State: Initialized, Role: FOLLOWER
I20250903 08:08:49.902750 21820 consensus_queue.cc:260] T 6b660cab5f9e41768b2074130cbe8a4d P 9bf6ef1f80704b49bf1b4354718df875 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5050fb31d1413a949c35c167d78324" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 34563 } } peers { permanent_uuid: "9bf6ef1f80704b49bf1b4354718df875" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 46253 } } peers { permanent_uuid: "dc83179923a84bfaa56be4f55a7158ef" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 36249 } }
I20250903 08:08:49.903028 21820 ts_tablet_manager.cc:1428] T 6b660cab5f9e41768b2074130cbe8a4d P 9bf6ef1f80704b49bf1b4354718df875: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250903 08:08:49.903362 21285 catalog_manager.cc:4928] TS dc83179923a84bfaa56be4f55a7158ef (127.12.201.195:36249): tablet 321081ef9a2649069b101640791cee3c (table test-table [id=8c85b7eb0bb84ca58fc48728b152663f]) successfully deleted
W20250903 08:08:49.931082 21791 tablet.cc:2378] T 6b660cab5f9e41768b2074130cbe8a4d P dc83179923a84bfaa56be4f55a7158ef: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250903 08:08:49.951714 21649 tablet.cc:2378] T 6b660cab5f9e41768b2074130cbe8a4d P 9bf6ef1f80704b49bf1b4354718df875: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250903 08:08:49.951880 21826 raft_consensus.cc:491] T 6b660cab5f9e41768b2074130cbe8a4d P 9bf6ef1f80704b49bf1b4354718df875 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 08:08:49.952011 21826 raft_consensus.cc:513] T 6b660cab5f9e41768b2074130cbe8a4d P 9bf6ef1f80704b49bf1b4354718df875 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5050fb31d1413a949c35c167d78324" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 34563 } } peers { permanent_uuid: "9bf6ef1f80704b49bf1b4354718df875" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 46253 } } peers { permanent_uuid: "dc83179923a84bfaa56be4f55a7158ef" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 36249 } }
I20250903 08:08:49.953224 21826 leader_election.cc:290] T 6b660cab5f9e41768b2074130cbe8a4d P 9bf6ef1f80704b49bf1b4354718df875 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa5050fb31d1413a949c35c167d78324 (127.12.201.193:34563), dc83179923a84bfaa56be4f55a7158ef (127.12.201.195:36249)
I20250903 08:08:49.956732 21848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:49.953415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.193:34563 (local address 127.12.201.194:54253)
0903 08:08:49.953861 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:49.953877 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:49.954012 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:49.954612 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:49.954616 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:49.954634 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 08:08:49.954854 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:49.954860 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:49.955632 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:49.955638 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:49.956301 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:49.956310 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:49.956425 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:49.956449 (+ 24us) client_negotiation.cc:769] Sending connection context
0903 08:08:49.956503 (+ 54us) client_negotiation.cc:240] Negotiation successful
0903 08:08:49.956566 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"spinlock_wait_cycles":4224,"thread_start_us":82,"threads_started":1}
I20250903 08:08:49.957230 21816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:49.953575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.194:54253 (local address 127.12.201.193:34563)
0903 08:08:49.953706 (+ 131us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:49.953711 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:49.953962 (+ 251us) server_negotiation.cc:408] Connection header received
0903 08:08:49.954413 (+ 451us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:49.954417 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:49.954475 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:49.954549 (+ 74us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 08:08:49.955004 (+ 455us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:49.955511 (+ 507us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:49.956419 (+ 908us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:49.956916 (+ 497us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:49.956968 (+ 52us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:49.957024 (+ 56us) server_negotiation.cc:300] Negotiation successful
0903 08:08:49.957090 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20250903 08:08:49.957762 21815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:49.953584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.194:59097 (local address 127.12.201.195:36249)
0903 08:08:49.953818 (+ 234us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:49.953823 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:49.953992 (+ 169us) server_negotiation.cc:408] Connection header received
0903 08:08:49.954114 (+ 122us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:49.954118 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:49.954185 (+ 67us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:49.954279 (+ 94us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 08:08:49.954739 (+ 460us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:49.955539 (+ 800us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:49.956782 (+ 1243us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:49.957240 (+ 458us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:49.957274 (+ 34us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:49.957585 (+ 311us) server_negotiation.cc:300] Negotiation successful
0903 08:08:49.957656 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149}
I20250903 08:08:49.957875 21446 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b660cab5f9e41768b2074130cbe8a4d" candidate_uuid: "9bf6ef1f80704b49bf1b4354718df875" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa5050fb31d1413a949c35c167d78324" is_pre_election: true
I20250903 08:08:49.958101 21446 raft_consensus.cc:2466] T 6b660cab5f9e41768b2074130cbe8a4d P fa5050fb31d1413a949c35c167d78324 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bf6ef1f80704b49bf1b4354718df875 in term 0.
I20250903 08:08:49.957638 21847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:49.953420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.195:36249 (local address 127.12.201.194:59097)
0903 08:08:49.953863 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:49.953877 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:49.954012 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:49.954326 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:49.954329 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:49.954350 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 08:08:49.954615 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:49.954623 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:49.955702 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:49.955707 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:49.956636 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:49.956646 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:49.957331 (+ 685us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:49.957347 (+ 16us) client_negotiation.cc:769] Sending connection context
0903 08:08:49.957427 (+ 80us) client_negotiation.cc:240] Negotiation successful
0903 08:08:49.957493 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"mutex_wait_us":212,"spinlock_wait_cycles":10368,"thread_start_us":58,"threads_started":1}
I20250903 08:08:49.958492 21536 leader_election.cc:304] T 6b660cab5f9e41768b2074130cbe8a4d P 9bf6ef1f80704b49bf1b4354718df875 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bf6ef1f80704b49bf1b4354718df875, fa5050fb31d1413a949c35c167d78324; no voters:
I20250903 08:08:49.958741 21826 raft_consensus.cc:2802] T 6b660cab5f9e41768b2074130cbe8a4d P 9bf6ef1f80704b49bf1b4354718df875 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 08:08:49.958802 21826 raft_consensus.cc:491] T 6b660cab5f9e41768b2074130cbe8a4d P 9bf6ef1f80704b49bf1b4354718df875 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 08:08:49.958842 21826 raft_consensus.cc:3058] T 6b660cab5f9e41768b2074130cbe8a4d P 9bf6ef1f80704b49bf1b4354718df875 [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:08:49.958879 21744 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b660cab5f9e41768b2074130cbe8a4d" candidate_uuid: "9bf6ef1f80704b49bf1b4354718df875" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc83179923a84bfaa56be4f55a7158ef" is_pre_election: true
I20250903 08:08:49.959012 21744 raft_consensus.cc:2466] T 6b660cab5f9e41768b2074130cbe8a4d P dc83179923a84bfaa56be4f55a7158ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bf6ef1f80704b49bf1b4354718df875 in term 0.
I20250903 08:08:49.959995 21826 raft_consensus.cc:513] T 6b660cab5f9e41768b2074130cbe8a4d P 9bf6ef1f80704b49bf1b4354718df875 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5050fb31d1413a949c35c167d78324" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 34563 } } peers { permanent_uuid: "9bf6ef1f80704b49bf1b4354718df875" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 46253 } } peers { permanent_uuid: "dc83179923a84bfaa56be4f55a7158ef" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 36249 } }
I20250903 08:08:49.960394 21826 leader_election.cc:290] T 6b660cab5f9e41768b2074130cbe8a4d P 9bf6ef1f80704b49bf1b4354718df875 [CANDIDATE]: Term 1 election: Requested vote from peers fa5050fb31d1413a949c35c167d78324 (127.12.201.193:34563), dc83179923a84bfaa56be4f55a7158ef (127.12.201.195:36249)
I20250903 08:08:49.960702 21446 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b660cab5f9e41768b2074130cbe8a4d" candidate_uuid: "9bf6ef1f80704b49bf1b4354718df875" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa5050fb31d1413a949c35c167d78324"
I20250903 08:08:49.960697 21744 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b660cab5f9e41768b2074130cbe8a4d" candidate_uuid: "9bf6ef1f80704b49bf1b4354718df875" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc83179923a84bfaa56be4f55a7158ef"
I20250903 08:08:49.960819 21744 raft_consensus.cc:3058] T 6b660cab5f9e41768b2074130cbe8a4d P dc83179923a84bfaa56be4f55a7158ef [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:08:49.960819 21446 raft_consensus.cc:3058] T 6b660cab5f9e41768b2074130cbe8a4d P fa5050fb31d1413a949c35c167d78324 [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:08:49.961913 21744 raft_consensus.cc:2466] T 6b660cab5f9e41768b2074130cbe8a4d P dc83179923a84bfaa56be4f55a7158ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bf6ef1f80704b49bf1b4354718df875 in term 1.
I20250903 08:08:49.961913 21446 raft_consensus.cc:2466] T 6b660cab5f9e41768b2074130cbe8a4d P fa5050fb31d1413a949c35c167d78324 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bf6ef1f80704b49bf1b4354718df875 in term 1.
I20250903 08:08:49.962219 21533 leader_election.cc:304] T 6b660cab5f9e41768b2074130cbe8a4d P 9bf6ef1f80704b49bf1b4354718df875 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bf6ef1f80704b49bf1b4354718df875, dc83179923a84bfaa56be4f55a7158ef; no voters:
I20250903 08:08:49.962522 21826 raft_consensus.cc:2802] T 6b660cab5f9e41768b2074130cbe8a4d P 9bf6ef1f80704b49bf1b4354718df875 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 08:08:49.962690 21826 raft_consensus.cc:695] T 6b660cab5f9e41768b2074130cbe8a4d P 9bf6ef1f80704b49bf1b4354718df875 [term 1 LEADER]: Becoming Leader. State: Replica: 9bf6ef1f80704b49bf1b4354718df875, State: Running, Role: LEADER
I20250903 08:08:49.962957 21826 consensus_queue.cc:237] T 6b660cab5f9e41768b2074130cbe8a4d P 9bf6ef1f80704b49bf1b4354718df875 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5050fb31d1413a949c35c167d78324" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 34563 } } peers { permanent_uuid: "9bf6ef1f80704b49bf1b4354718df875" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 46253 } } peers { permanent_uuid: "dc83179923a84bfaa56be4f55a7158ef" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 36249 } }
I20250903 08:08:49.965260 21301 catalog_manager.cc:5582] T 6b660cab5f9e41768b2074130cbe8a4d P 9bf6ef1f80704b49bf1b4354718df875 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9bf6ef1f80704b49bf1b4354718df875 (127.12.201.194). New cstate: current_term: 1 leader_uuid: "9bf6ef1f80704b49bf1b4354718df875" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5050fb31d1413a949c35c167d78324" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 34563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9bf6ef1f80704b49bf1b4354718df875" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 46253 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc83179923a84bfaa56be4f55a7158ef" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 36249 } health_report { overall_health: UNKNOWN } } }
W20250903 08:08:49.980855 21507 tablet.cc:2378] T 6b660cab5f9e41768b2074130cbe8a4d P fa5050fb31d1413a949c35c167d78324: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 03 08:08:50 dist-test-slave-27r9 krb5kdc[21253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886928, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.201.254@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM
I20250903 08:08:50.012555 21350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:50.004792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45676 (local address 127.12.201.254:38257)
0903 08:08:50.004930 (+ 138us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:50.004937 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:50.005196 (+ 259us) server_negotiation.cc:408] Connection header received
0903 08:08:50.005333 (+ 137us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:50.005336 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:50.005376 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:50.005442 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:50.006200 (+ 758us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:50.006694 (+ 494us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:50.007314 (+ 620us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:50.007488 (+ 174us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:50.009491 (+ 2003us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:50.009508 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:50.009511 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:08:50.009532 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:50.011056 (+ 1524us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:50.011481 (+ 425us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:50.011486 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:50.011488 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:50.011530 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:50.011729 (+ 199us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:50.011732 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:50.011734 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:50.011876 (+ 142us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:50.011963 (+ 87us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:50.012218 (+ 255us) server_negotiation.cc:300] Negotiation successful
0903 08:08:50.012390 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":63,"server-negotiator.queue_time_us":42}
I20250903 08:08:50.012557 21856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:50.004751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:38257 (local address 127.0.0.1:45676)
0903 08:08:50.005100 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:50.005115 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:50.005243 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:50.005470 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:50.005474 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:50.005889 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:08:50.006078 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:50.006085 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:50.006797 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:50.006801 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:50.007209 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:50.007217 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:50.007329 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:50.007964 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:08:50.007977 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:08:50.009369 (+ 1392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:08:50.011153 (+ 1784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:50.011163 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:50.011169 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:50.011384 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:50.011608 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:50.011611 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:50.011615 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:50.011656 (+ 41us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:50.011969 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:08:50.011978 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:08:50.012069 (+ 91us) client_negotiation.cc:769] Sending connection context
0903 08:08:50.012168 (+ 99us) client_negotiation.cc:240] Negotiation successful
0903 08:08:50.012323 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":109,"threads_started":1}
I20250903 08:08:50.016906 21301 catalog_manager.cc:2232] Servicing CreateTable request from {username='oryx', principal='oryx/127.12.201.254@KRBTEST.COM'} at 127.0.0.1:45676:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20250903 08:08:50.017570 21301 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 08:08:50.022257 21440 tablet_service.cc:1468] Processing CreateTablet for tablet 42661f4801c442839cc640a9f6067fac (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f86e03aa57aa4aad81b0596bb7be9544]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20250903 08:08:50.022442 21582 tablet_service.cc:1468] Processing CreateTablet for tablet 42661f4801c442839cc640a9f6067fac (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f86e03aa57aa4aad81b0596bb7be9544]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20250903 08:08:50.022678 21440 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42661f4801c442839cc640a9f6067fac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:08:50.022773 21582 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42661f4801c442839cc640a9f6067fac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:08:50.025049 21820 tablet_bootstrap.cc:492] T 42661f4801c442839cc640a9f6067fac P 9bf6ef1f80704b49bf1b4354718df875: Bootstrap starting.
I20250903 08:08:50.025422 21818 tablet_bootstrap.cc:492] T 42661f4801c442839cc640a9f6067fac P fa5050fb31d1413a949c35c167d78324: Bootstrap starting.
I20250903 08:08:50.025899 21724 tablet_service.cc:1468] Processing CreateTablet for tablet 42661f4801c442839cc640a9f6067fac (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f86e03aa57aa4aad81b0596bb7be9544]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20250903 08:08:50.026069 21820 tablet_bootstrap.cc:654] T 42661f4801c442839cc640a9f6067fac P 9bf6ef1f80704b49bf1b4354718df875: Neither blocks nor log segments found. Creating new log.
I20250903 08:08:50.026271 21724 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42661f4801c442839cc640a9f6067fac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:08:50.026517 21818 tablet_bootstrap.cc:654] T 42661f4801c442839cc640a9f6067fac P fa5050fb31d1413a949c35c167d78324: Neither blocks nor log segments found. Creating new log.
I20250903 08:08:50.027657 21818 tablet_bootstrap.cc:492] T 42661f4801c442839cc640a9f6067fac P fa5050fb31d1413a949c35c167d78324: No bootstrap required, opened a new log
I20250903 08:08:50.027657 21820 tablet_bootstrap.cc:492] T 42661f4801c442839cc640a9f6067fac P 9bf6ef1f80704b49bf1b4354718df875: No bootstrap required, opened a new log
I20250903 08:08:50.027731 21818 ts_tablet_manager.cc:1397] T 42661f4801c442839cc640a9f6067fac P fa5050fb31d1413a949c35c167d78324: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20250903 08:08:50.027731 21820 ts_tablet_manager.cc:1397] T 42661f4801c442839cc640a9f6067fac P 9bf6ef1f80704b49bf1b4354718df875: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20250903 08:08:50.028193 21820 raft_consensus.cc:357] T 42661f4801c442839cc640a9f6067fac P 9bf6ef1f80704b49bf1b4354718df875 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5050fb31d1413a949c35c167d78324" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 34563 } } peers { permanent_uuid: "9bf6ef1f80704b49bf1b4354718df875" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 46253 } } peers { permanent_uuid: "dc83179923a84bfaa56be4f55a7158ef" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 36249 } }
I20250903 08:08:50.028193 21818 raft_consensus.cc:357] T 42661f4801c442839cc640a9f6067fac P fa5050fb31d1413a949c35c167d78324 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5050fb31d1413a949c35c167d78324" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 34563 } } peers { permanent_uuid: "9bf6ef1f80704b49bf1b4354718df875" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 46253 } } peers { permanent_uuid: "dc83179923a84bfaa56be4f55a7158ef" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 36249 } }
I20250903 08:08:50.028307 21820 raft_consensus.cc:383] T 42661f4801c442839cc640a9f6067fac P 9bf6ef1f80704b49bf1b4354718df875 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:08:50.028307 21818 raft_consensus.cc:383] T 42661f4801c442839cc640a9f6067fac P fa5050fb31d1413a949c35c167d78324 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:08:50.028345 21820 raft_consensus.cc:738] T 42661f4801c442839cc640a9f6067fac P 9bf6ef1f80704b49bf1b4354718df875 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bf6ef1f80704b49bf1b4354718df875, State: Initialized, Role: FOLLOWER
I20250903 08:08:50.028345 21818 raft_consensus.cc:738] T 42661f4801c442839cc640a9f6067fac P fa5050fb31d1413a949c35c167d78324 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa5050fb31d1413a949c35c167d78324, State: Initialized, Role: FOLLOWER
I20250903 08:08:50.028470 21818 consensus_queue.cc:260] T 42661f4801c442839cc640a9f6067fac P fa5050fb31d1413a949c35c167d78324 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5050fb31d1413a949c35c167d78324" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 34563 } } peers { permanent_uuid: "9bf6ef1f80704b49bf1b4354718df875" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 46253 } } peers { permanent_uuid: "dc83179923a84bfaa56be4f55a7158ef" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 36249 } }
I20250903 08:08:50.028470 21820 consensus_queue.cc:260] T 42661f4801c442839cc640a9f6067fac P 9bf6ef1f80704b49bf1b4354718df875 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5050fb31d1413a949c35c167d78324" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 34563 } } peers { permanent_uuid: "9bf6ef1f80704b49bf1b4354718df875" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 46253 } } peers { permanent_uuid: "dc83179923a84bfaa56be4f55a7158ef" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 36249 } }
I20250903 08:08:50.028728 21818 ts_tablet_manager.cc:1428] T 42661f4801c442839cc640a9f6067fac P fa5050fb31d1413a949c35c167d78324: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250903 08:08:50.028754 21820 ts_tablet_manager.cc:1428] T 42661f4801c442839cc640a9f6067fac P 9bf6ef1f80704b49bf1b4354718df875: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250903 08:08:50.028712 21826 tablet_replica.cc:440] T 42661f4801c442839cc640a9f6067fac P 9bf6ef1f80704b49bf1b4354718df875: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5050fb31d1413a949c35c167d78324" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 34563 } } peers { permanent_uuid: "9bf6ef1f80704b49bf1b4354718df875" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 46253 } } peers { permanent_uuid: "dc83179923a84bfaa56be4f55a7158ef" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 36249 } } }
I20250903 08:08:50.028836 21826 tablet_replica.cc:443] T 42661f4801c442839cc640a9f6067fac P 9bf6ef1f80704b49bf1b4354718df875: This TxnStatusTablet replica's current role is: FOLLOWER
I20250903 08:08:50.028915 21825 tablet_replica.cc:440] T 42661f4801c442839cc640a9f6067fac P fa5050fb31d1413a949c35c167d78324: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5050fb31d1413a949c35c167d78324" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 34563 } } peers { permanent_uuid: "9bf6ef1f80704b49bf1b4354718df875" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 46253 } } peers { permanent_uuid: "dc83179923a84bfaa56be4f55a7158ef" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 36249 } } }
I20250903 08:08:50.029008 21825 tablet_replica.cc:443] T 42661f4801c442839cc640a9f6067fac P fa5050fb31d1413a949c35c167d78324: This TxnStatusTablet replica's current role is: FOLLOWER
I20250903 08:08:50.030673 21819 tablet_bootstrap.cc:492] T 42661f4801c442839cc640a9f6067fac P dc83179923a84bfaa56be4f55a7158ef: Bootstrap starting.
I20250903 08:08:50.031723 21819 tablet_bootstrap.cc:654] T 42661f4801c442839cc640a9f6067fac P dc83179923a84bfaa56be4f55a7158ef: Neither blocks nor log segments found. Creating new log.
I20250903 08:08:50.033124 21819 tablet_bootstrap.cc:492] T 42661f4801c442839cc640a9f6067fac P dc83179923a84bfaa56be4f55a7158ef: No bootstrap required, opened a new log
I20250903 08:08:50.033221 21819 ts_tablet_manager.cc:1397] T 42661f4801c442839cc640a9f6067fac P dc83179923a84bfaa56be4f55a7158ef: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250903 08:08:50.033700 21819 raft_consensus.cc:357] T 42661f4801c442839cc640a9f6067fac P dc83179923a84bfaa56be4f55a7158ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5050fb31d1413a949c35c167d78324" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 34563 } } peers { permanent_uuid: "9bf6ef1f80704b49bf1b4354718df875" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 46253 } } peers { permanent_uuid: "dc83179923a84bfaa56be4f55a7158ef" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 36249 } }
I20250903 08:08:50.033820 21819 raft_consensus.cc:383] T 42661f4801c442839cc640a9f6067fac P dc83179923a84bfaa56be4f55a7158ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:08:50.033864 21819 raft_consensus.cc:738] T 42661f4801c442839cc640a9f6067fac P dc83179923a84bfaa56be4f55a7158ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc83179923a84bfaa56be4f55a7158ef, State: Initialized, Role: FOLLOWER
I20250903 08:08:50.033994 21819 consensus_queue.cc:260] T 42661f4801c442839cc640a9f6067fac P dc83179923a84bfaa56be4f55a7158ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5050fb31d1413a949c35c167d78324" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 34563 } } peers { permanent_uuid: "9bf6ef1f80704b49bf1b4354718df875" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 46253 } } peers { permanent_uuid: "dc83179923a84bfaa56be4f55a7158ef" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 36249 } }
I20250903 08:08:50.034268 21824 tablet_replica.cc:440] T 42661f4801c442839cc640a9f6067fac P dc83179923a84bfaa56be4f55a7158ef: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5050fb31d1413a949c35c167d78324" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 34563 } } peers { permanent_uuid: "9bf6ef1f80704b49bf1b4354718df875" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 46253 } } peers { permanent_uuid: "dc83179923a84bfaa56be4f55a7158ef" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 36249 } } }
I20250903 08:08:50.034376 21824 tablet_replica.cc:443] T 42661f4801c442839cc640a9f6067fac P dc83179923a84bfaa56be4f55a7158ef: This TxnStatusTablet replica's current role is: FOLLOWER
I20250903 08:08:50.034679 21819 ts_tablet_manager.cc:1428] T 42661f4801c442839cc640a9f6067fac P dc83179923a84bfaa56be4f55a7158ef: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250903 08:08:50.115516 21824 raft_consensus.cc:491] T 42661f4801c442839cc640a9f6067fac P dc83179923a84bfaa56be4f55a7158ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 08:08:50.115666 21824 raft_consensus.cc:513] T 42661f4801c442839cc640a9f6067fac P dc83179923a84bfaa56be4f55a7158ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5050fb31d1413a949c35c167d78324" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 34563 } } peers { permanent_uuid: "9bf6ef1f80704b49bf1b4354718df875" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 46253 } } peers { permanent_uuid: "dc83179923a84bfaa56be4f55a7158ef" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 36249 } }
I20250903 08:08:50.116679 21824 leader_election.cc:290] T 42661f4801c442839cc640a9f6067fac P dc83179923a84bfaa56be4f55a7158ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa5050fb31d1413a949c35c167d78324 (127.12.201.193:34563), 9bf6ef1f80704b49bf1b4354718df875 (127.12.201.194:46253)
I20250903 08:08:50.119467 21683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:50.116977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.194:46253 (local address 127.12.201.195:52601)
0903 08:08:50.117107 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:50.117125 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:50.117256 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:50.117503 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:50.117506 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:50.117522 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 08:08:50.117719 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:50.117725 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:50.118464 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:50.118467 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:50.119076 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:50.119084 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:50.119193 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:50.119215 (+ 22us) client_negotiation.cc:769] Sending connection context
0903 08:08:50.119280 (+ 65us) client_negotiation.cc:240] Negotiation successful
0903 08:08:50.119338 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20250903 08:08:50.119891 21814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:50.117057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.195:52601 (local address 127.12.201.194:46253)
0903 08:08:50.117195 (+ 138us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:50.117199 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:50.117214 (+ 15us) server_negotiation.cc:408] Connection header received
0903 08:08:50.117367 (+ 153us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:50.117371 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:50.117418 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:50.117500 (+ 82us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 08:08:50.117832 (+ 332us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:50.118339 (+ 507us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:50.119192 (+ 853us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:50.119664 (+ 472us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:50.119711 (+ 47us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:50.119762 (+ 51us) server_negotiation.cc:300] Negotiation successful
0903 08:08:50.119808 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20250903 08:08:50.120299 21602 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "42661f4801c442839cc640a9f6067fac" candidate_uuid: "dc83179923a84bfaa56be4f55a7158ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bf6ef1f80704b49bf1b4354718df875" is_pre_election: true
I20250903 08:08:50.120469 21816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:50.117104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.195:35115 (local address 127.12.201.193:34563)
0903 08:08:50.117226 (+ 122us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:50.117231 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:50.117619 (+ 388us) server_negotiation.cc:408] Connection header received
0903 08:08:50.117758 (+ 139us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:50.117760 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:50.117834 (+ 74us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:50.117907 (+ 73us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 08:08:50.118270 (+ 363us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:50.118781 (+ 511us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:50.119667 (+ 886us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:50.120060 (+ 393us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:50.120101 (+ 41us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:50.120299 (+ 198us) server_negotiation.cc:300] Negotiation successful
0903 08:08:50.120363 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20250903 08:08:50.120483 21860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:50.116977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.193:34563 (local address 127.12.201.195:35115)
0903 08:08:50.117528 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:50.117539 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:50.117643 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:50.117919 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:50.117922 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:50.117939 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 08:08:50.118164 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:50.118170 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:50.118884 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:50.118889 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:50.119513 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:50.119525 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:50.120159 (+ 634us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:50.120178 (+ 19us) client_negotiation.cc:769] Sending connection context
0903 08:08:50.120261 (+ 83us) client_negotiation.cc:240] Negotiation successful
0903 08:08:50.120317 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"mutex_wait_us":210,"thread_start_us":136,"threads_started":1}
I20250903 08:08:50.120471 21602 raft_consensus.cc:2466] T 42661f4801c442839cc640a9f6067fac P 9bf6ef1f80704b49bf1b4354718df875 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc83179923a84bfaa56be4f55a7158ef in term 0.
I20250903 08:08:50.120860 21676 leader_election.cc:304] T 42661f4801c442839cc640a9f6067fac P dc83179923a84bfaa56be4f55a7158ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bf6ef1f80704b49bf1b4354718df875, dc83179923a84bfaa56be4f55a7158ef; no voters:
I20250903 08:08:50.121110 21824 raft_consensus.cc:2802] T 42661f4801c442839cc640a9f6067fac P dc83179923a84bfaa56be4f55a7158ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 08:08:50.121086 21446 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "42661f4801c442839cc640a9f6067fac" candidate_uuid: "dc83179923a84bfaa56be4f55a7158ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa5050fb31d1413a949c35c167d78324" is_pre_election: true
I20250903 08:08:50.121162 21824 raft_consensus.cc:491] T 42661f4801c442839cc640a9f6067fac P dc83179923a84bfaa56be4f55a7158ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 08:08:50.121191 21824 raft_consensus.cc:3058] T 42661f4801c442839cc640a9f6067fac P dc83179923a84bfaa56be4f55a7158ef [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:08:50.121219 21446 raft_consensus.cc:2466] T 42661f4801c442839cc640a9f6067fac P fa5050fb31d1413a949c35c167d78324 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc83179923a84bfaa56be4f55a7158ef in term 0.
I20250903 08:08:50.122119 21824 raft_consensus.cc:513] T 42661f4801c442839cc640a9f6067fac P dc83179923a84bfaa56be4f55a7158ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5050fb31d1413a949c35c167d78324" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 34563 } } peers { permanent_uuid: "9bf6ef1f80704b49bf1b4354718df875" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 46253 } } peers { permanent_uuid: "dc83179923a84bfaa56be4f55a7158ef" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 36249 } }
I20250903 08:08:50.122488 21824 leader_election.cc:290] T 42661f4801c442839cc640a9f6067fac P dc83179923a84bfaa56be4f55a7158ef [CANDIDATE]: Term 1 election: Requested vote from peers fa5050fb31d1413a949c35c167d78324 (127.12.201.193:34563), 9bf6ef1f80704b49bf1b4354718df875 (127.12.201.194:46253)
I20250903 08:08:50.122830 21446 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "42661f4801c442839cc640a9f6067fac" candidate_uuid: "dc83179923a84bfaa56be4f55a7158ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa5050fb31d1413a949c35c167d78324"
I20250903 08:08:50.122865 21602 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "42661f4801c442839cc640a9f6067fac" candidate_uuid: "dc83179923a84bfaa56be4f55a7158ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bf6ef1f80704b49bf1b4354718df875"
I20250903 08:08:50.122957 21446 raft_consensus.cc:3058] T 42661f4801c442839cc640a9f6067fac P fa5050fb31d1413a949c35c167d78324 [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:08:50.122997 21602 raft_consensus.cc:3058] T 42661f4801c442839cc640a9f6067fac P 9bf6ef1f80704b49bf1b4354718df875 [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:08:50.124056 21602 raft_consensus.cc:2466] T 42661f4801c442839cc640a9f6067fac P 9bf6ef1f80704b49bf1b4354718df875 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc83179923a84bfaa56be4f55a7158ef in term 1.
I20250903 08:08:50.124056 21446 raft_consensus.cc:2466] T 42661f4801c442839cc640a9f6067fac P fa5050fb31d1413a949c35c167d78324 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc83179923a84bfaa56be4f55a7158ef in term 1.
I20250903 08:08:50.124388 21676 leader_election.cc:304] T 42661f4801c442839cc640a9f6067fac P dc83179923a84bfaa56be4f55a7158ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bf6ef1f80704b49bf1b4354718df875, dc83179923a84bfaa56be4f55a7158ef; no voters:
I20250903 08:08:50.124600 21824 raft_consensus.cc:2802] T 42661f4801c442839cc640a9f6067fac P dc83179923a84bfaa56be4f55a7158ef [term 1 FOLLOWER]: Leader election won for term 1
I20250903 08:08:50.124791 21824 raft_consensus.cc:695] T 42661f4801c442839cc640a9f6067fac P dc83179923a84bfaa56be4f55a7158ef [term 1 LEADER]: Becoming Leader. State: Replica: dc83179923a84bfaa56be4f55a7158ef, State: Running, Role: LEADER
I20250903 08:08:50.125041 21824 consensus_queue.cc:237] T 42661f4801c442839cc640a9f6067fac P dc83179923a84bfaa56be4f55a7158ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5050fb31d1413a949c35c167d78324" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 34563 } } peers { permanent_uuid: "9bf6ef1f80704b49bf1b4354718df875" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 46253 } } peers { permanent_uuid: "dc83179923a84bfaa56be4f55a7158ef" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 36249 } }
I20250903 08:08:50.126112 21861 tablet_replica.cc:440] T 42661f4801c442839cc640a9f6067fac P dc83179923a84bfaa56be4f55a7158ef: TxnStatusTablet state changed. Reason: New leader dc83179923a84bfaa56be4f55a7158ef. Latest consensus state: current_term: 1 leader_uuid: "dc83179923a84bfaa56be4f55a7158ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5050fb31d1413a949c35c167d78324" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 34563 } } peers { permanent_uuid: "9bf6ef1f80704b49bf1b4354718df875" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 46253 } } peers { permanent_uuid: "dc83179923a84bfaa56be4f55a7158ef" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 36249 } } }
I20250903 08:08:50.126189 21861 tablet_replica.cc:443] T 42661f4801c442839cc640a9f6067fac P dc83179923a84bfaa56be4f55a7158ef: This TxnStatusTablet replica's current role is: LEADER
I20250903 08:08:50.126644 21863 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20250903 08:08:50.126765 21863 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20250903 08:08:50.127499 21301 catalog_manager.cc:5582] T 42661f4801c442839cc640a9f6067fac P dc83179923a84bfaa56be4f55a7158ef reported cstate change: term changed from 0 to 1, leader changed from <none> to dc83179923a84bfaa56be4f55a7158ef (127.12.201.195). New cstate: current_term: 1 leader_uuid: "dc83179923a84bfaa56be4f55a7158ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5050fb31d1413a949c35c167d78324" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 34563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9bf6ef1f80704b49bf1b4354718df875" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 46253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc83179923a84bfaa56be4f55a7158ef" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 36249 } health_report { overall_health: HEALTHY } } }
I20250903 08:08:50.160755 21815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:50.157704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42226 (local address 127.12.201.195:36249)
0903 08:08:50.157841 (+ 137us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:50.157845 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:50.157859 (+ 14us) server_negotiation.cc:408] Connection header received
0903 08:08:50.158065 (+ 206us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:50.158069 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:50.158127 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:50.158205 (+ 78us) server_negotiation.cc:227] Negotiated authn=TOKEN
0903 08:08:50.158558 (+ 353us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:50.159060 (+ 502us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:50.159692 (+ 632us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:50.159877 (+ 185us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:50.159928 (+ 51us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0903 08:08:50.160354 (+ 426us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0903 08:08:50.160458 (+ 104us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:50.160568 (+ 110us) server_negotiation.cc:300] Negotiation successful
0903 08:08:50.160615 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20250903 08:08:50.160755 21856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:50.157617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.195:36249 (local address 127.0.0.1:42226)
0903 08:08:50.157768 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:50.157785 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:50.157926 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:50.158221 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:50.158224 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:50.158232 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0903 08:08:50.158432 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:50.158438 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:50.159164 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:50.159168 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:50.159579 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:50.159586 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:50.159690 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:50.159736 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0903 08:08:50.160452 (+ 716us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0903 08:08:50.160470 (+ 18us) client_negotiation.cc:769] Sending connection context
0903 08:08:50.160545 (+ 75us) client_negotiation.cc:240] Negotiation successful
0903 08:08:50.160625 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20250903 08:08:50.163709 21602 raft_consensus.cc:1273] T 42661f4801c442839cc640a9f6067fac P 9bf6ef1f80704b49bf1b4354718df875 [term 1 FOLLOWER]: Refusing update from remote peer dc83179923a84bfaa56be4f55a7158ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 08:08:50.163916 21446 raft_consensus.cc:1273] T 42661f4801c442839cc640a9f6067fac P fa5050fb31d1413a949c35c167d78324 [term 1 FOLLOWER]: Refusing update from remote peer dc83179923a84bfaa56be4f55a7158ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 08:08:50.164158 21861 consensus_queue.cc:1035] T 42661f4801c442839cc640a9f6067fac P dc83179923a84bfaa56be4f55a7158ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bf6ef1f80704b49bf1b4354718df875" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 46253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 08:08:50.164419 21824 consensus_queue.cc:1035] T 42661f4801c442839cc640a9f6067fac P dc83179923a84bfaa56be4f55a7158ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa5050fb31d1413a949c35c167d78324" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 08:08:50.166013 21826 tablet_replica.cc:440] T 42661f4801c442839cc640a9f6067fac P 9bf6ef1f80704b49bf1b4354718df875: TxnStatusTablet state changed. Reason: New leader dc83179923a84bfaa56be4f55a7158ef. Latest consensus state: current_term: 1 leader_uuid: "dc83179923a84bfaa56be4f55a7158ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5050fb31d1413a949c35c167d78324" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 34563 } } peers { permanent_uuid: "9bf6ef1f80704b49bf1b4354718df875" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 46253 } } peers { permanent_uuid: "dc83179923a84bfaa56be4f55a7158ef" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 36249 } } }
I20250903 08:08:50.166121 21826 tablet_replica.cc:443] T 42661f4801c442839cc640a9f6067fac P 9bf6ef1f80704b49bf1b4354718df875: This TxnStatusTablet replica's current role is: FOLLOWER
I20250903 08:08:50.166772 21825 tablet_replica.cc:440] T 42661f4801c442839cc640a9f6067fac P fa5050fb31d1413a949c35c167d78324: TxnStatusTablet state changed. Reason: New leader dc83179923a84bfaa56be4f55a7158ef. Latest consensus state: current_term: 1 leader_uuid: "dc83179923a84bfaa56be4f55a7158ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5050fb31d1413a949c35c167d78324" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 34563 } } peers { permanent_uuid: "9bf6ef1f80704b49bf1b4354718df875" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 46253 } } peers { permanent_uuid: "dc83179923a84bfaa56be4f55a7158ef" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 36249 } } }
I20250903 08:08:50.166877 21825 tablet_replica.cc:443] T 42661f4801c442839cc640a9f6067fac P fa5050fb31d1413a949c35c167d78324: This TxnStatusTablet replica's current role is: FOLLOWER
I20250903 08:08:50.167502 21824 tablet_replica.cc:440] T 42661f4801c442839cc640a9f6067fac P dc83179923a84bfaa56be4f55a7158ef: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dc83179923a84bfaa56be4f55a7158ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5050fb31d1413a949c35c167d78324" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 34563 } } peers { permanent_uuid: "9bf6ef1f80704b49bf1b4354718df875" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 46253 } } peers { permanent_uuid: "dc83179923a84bfaa56be4f55a7158ef" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 36249 } } }
I20250903 08:08:50.167626 21824 tablet_replica.cc:443] T 42661f4801c442839cc640a9f6067fac P dc83179923a84bfaa56be4f55a7158ef: This TxnStatusTablet replica's current role is: LEADER
I20250903 08:08:50.168577 21826 tablet_replica.cc:440] T 42661f4801c442839cc640a9f6067fac P 9bf6ef1f80704b49bf1b4354718df875: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dc83179923a84bfaa56be4f55a7158ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5050fb31d1413a949c35c167d78324" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 34563 } } peers { permanent_uuid: "9bf6ef1f80704b49bf1b4354718df875" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 46253 } } peers { permanent_uuid: "dc83179923a84bfaa56be4f55a7158ef" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 36249 } } }
I20250903 08:08:50.168673 21826 tablet_replica.cc:443] T 42661f4801c442839cc640a9f6067fac P 9bf6ef1f80704b49bf1b4354718df875: This TxnStatusTablet replica's current role is: FOLLOWER
I20250903 08:08:50.173058 21861 tablet_replica.cc:440] T 42661f4801c442839cc640a9f6067fac P dc83179923a84bfaa56be4f55a7158ef: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dc83179923a84bfaa56be4f55a7158ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5050fb31d1413a949c35c167d78324" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 34563 } } peers { permanent_uuid: "9bf6ef1f80704b49bf1b4354718df875" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 46253 } } peers { permanent_uuid: "dc83179923a84bfaa56be4f55a7158ef" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 36249 } } }
I20250903 08:08:50.173162 21861 tablet_replica.cc:443] T 42661f4801c442839cc640a9f6067fac P dc83179923a84bfaa56be4f55a7158ef: This TxnStatusTablet replica's current role is: LEADER
I20250903 08:08:50.175297 21825 tablet_replica.cc:440] T 42661f4801c442839cc640a9f6067fac P fa5050fb31d1413a949c35c167d78324: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dc83179923a84bfaa56be4f55a7158ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5050fb31d1413a949c35c167d78324" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 34563 } } peers { permanent_uuid: "9bf6ef1f80704b49bf1b4354718df875" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 46253 } } peers { permanent_uuid: "dc83179923a84bfaa56be4f55a7158ef" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 36249 } } }
I20250903 08:08:50.175406 21825 tablet_replica.cc:443] T 42661f4801c442839cc640a9f6067fac P fa5050fb31d1413a949c35c167d78324: This TxnStatusTablet replica's current role is: FOLLOWER
I20250903 08:08:50.175962 21814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:50.171991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49340 (local address 127.12.201.194:46253)
0903 08:08:50.172128 (+ 137us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:50.172133 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:50.172300 (+ 167us) server_negotiation.cc:408] Connection header received
0903 08:08:50.172433 (+ 133us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:50.172436 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:50.172499 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:50.172600 (+ 101us) server_negotiation.cc:227] Negotiated authn=TOKEN
0903 08:08:50.173019 (+ 419us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:50.173662 (+ 643us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:50.174703 (+ 1041us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:50.174877 (+ 174us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:50.174941 (+ 64us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0903 08:08:50.175561 (+ 620us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0903 08:08:50.175671 (+ 110us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:50.175784 (+ 113us) server_negotiation.cc:300] Negotiation successful
0903 08:08:50.175835 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20250903 08:08:50.185235 21871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:50.182468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.195:36249 (local address 127.12.201.194:42187)
0903 08:08:50.182790 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:50.182806 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:50.182905 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:50.183167 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:50.183170 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:50.183178 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0903 08:08:50.183410 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:50.183415 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:50.184056 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:50.184059 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:50.184477 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:50.184483 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:50.184593 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:50.184634 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0903 08:08:50.184959 (+ 325us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0903 08:08:50.184965 (+ 6us) client_negotiation.cc:769] Sending connection context
0903 08:08:50.185034 (+ 69us) client_negotiation.cc:240] Negotiation successful
0903 08:08:50.185100 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":133,"threads_started":1}
I20250903 08:08:50.185222 21815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:50.182540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.194:42187 (local address 127.12.201.195:36249)
0903 08:08:50.182679 (+ 139us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:50.182683 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:50.182869 (+ 186us) server_negotiation.cc:408] Connection header received
0903 08:08:50.183035 (+ 166us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:50.183038 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:50.183085 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:50.183151 (+ 66us) server_negotiation.cc:227] Negotiated authn=TOKEN
0903 08:08:50.183517 (+ 366us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:50.183956 (+ 439us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:50.184591 (+ 635us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:50.184773 (+ 182us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:50.184812 (+ 39us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0903 08:08:50.184875 (+ 63us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0903 08:08:50.184941 (+ 66us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:50.185048 (+ 107us) server_negotiation.cc:300] Negotiation successful
0903 08:08:50.185090 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20250903 08:08:50.192026 21744 raft_consensus.cc:1273] T 6b660cab5f9e41768b2074130cbe8a4d P dc83179923a84bfaa56be4f55a7158ef [term 1 FOLLOWER]: Refusing update from remote peer 9bf6ef1f80704b49bf1b4354718df875: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 08:08:50.192113 21446 raft_consensus.cc:1273] T 6b660cab5f9e41768b2074130cbe8a4d P fa5050fb31d1413a949c35c167d78324 [term 1 FOLLOWER]: Refusing update from remote peer 9bf6ef1f80704b49bf1b4354718df875: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 08:08:50.192507 21826 consensus_queue.cc:1035] T 6b660cab5f9e41768b2074130cbe8a4d P 9bf6ef1f80704b49bf1b4354718df875 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa5050fb31d1413a949c35c167d78324" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 08:08:50.192672 21849 consensus_queue.cc:1035] T 6b660cab5f9e41768b2074130cbe8a4d P 9bf6ef1f80704b49bf1b4354718df875 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc83179923a84bfaa56be4f55a7158ef" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 36249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 08:08:50.213732 21877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:50.210374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.194:46253 (local address 127.12.201.195:33283)
0903 08:08:50.210710 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:50.210726 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:50.210833 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:50.211176 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:50.211180 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:50.211189 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0903 08:08:50.211483 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:50.211502 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:50.212410 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:50.212413 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:50.212950 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:50.212958 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:50.213047 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:50.213083 (+ 36us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0903 08:08:50.213494 (+ 411us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0903 08:08:50.213498 (+ 4us) client_negotiation.cc:769] Sending connection context
0903 08:08:50.213546 (+ 48us) client_negotiation.cc:240] Negotiation successful
0903 08:08:50.213605 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":119,"threads_started":1}
I20250903 08:08:50.213752 21814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:50.210462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.195:33283 (local address 127.12.201.194:46253)
0903 08:08:50.210594 (+ 132us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:50.210600 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:50.210796 (+ 196us) server_negotiation.cc:408] Connection header received
0903 08:08:50.210916 (+ 120us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:50.210918 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:50.210960 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:50.211059 (+ 99us) server_negotiation.cc:227] Negotiated authn=TOKEN
0903 08:08:50.211609 (+ 550us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:50.212267 (+ 658us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:50.213060 (+ 793us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:50.213266 (+ 206us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:50.213327 (+ 61us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0903 08:08:50.213404 (+ 77us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0903 08:08:50.213485 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:50.213571 (+ 86us) server_negotiation.cc:300] Negotiation successful
0903 08:08:50.213625 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20250903 08:08:50.237196 21301 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45672:
table { table_name: "test-table" } modify_external_catalogs: true
I20250903 08:08:50.237380 21301 catalog_manager.cc:2730] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45672:
table { table_name: "test-table" } modify_external_catalogs: true
I20250903 08:08:50.239768 21301 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3aa107cc15d641bca7f21dd4251fdf15: Sending DeleteTablet for 3 replicas of tablet 6b660cab5f9e41768b2074130cbe8a4d
I20250903 08:08:50.240315 21440 tablet_service.cc:1515] Processing DeleteTablet for tablet 6b660cab5f9e41768b2074130cbe8a4d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 08:08:50 UTC) from {username='oryx', principal='oryx/127.12.201.254@KRBTEST.COM'} at 127.0.0.1:54102
I20250903 08:08:50.240504 21842 tablet_replica.cc:331] T 6b660cab5f9e41768b2074130cbe8a4d P fa5050fb31d1413a949c35c167d78324: stopping tablet replica
I20250903 08:08:50.240638 21842 raft_consensus.cc:2241] T 6b660cab5f9e41768b2074130cbe8a4d P fa5050fb31d1413a949c35c167d78324 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 08:08:50.240758 21842 raft_consensus.cc:2270] T 6b660cab5f9e41768b2074130cbe8a4d P fa5050fb31d1413a949c35c167d78324 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 08:08:50.241217 21582 tablet_service.cc:1515] Processing DeleteTablet for tablet 6b660cab5f9e41768b2074130cbe8a4d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 08:08:50 UTC) from {username='oryx', principal='oryx/127.12.201.254@KRBTEST.COM'} at 127.0.0.1:49330
I20250903 08:08:50.241400 21841 tablet_replica.cc:331] T 6b660cab5f9e41768b2074130cbe8a4d P 9bf6ef1f80704b49bf1b4354718df875: stopping tablet replica
I20250903 08:08:50.241503 21841 raft_consensus.cc:2241] T 6b660cab5f9e41768b2074130cbe8a4d P 9bf6ef1f80704b49bf1b4354718df875 [term 1 LEADER]: Raft consensus shutting down.
I20250903 08:08:50.242002 21841 raft_consensus.cc:2270] T 6b660cab5f9e41768b2074130cbe8a4d P 9bf6ef1f80704b49bf1b4354718df875 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 08:08:50.242269 21724 tablet_service.cc:1515] Processing DeleteTablet for tablet 6b660cab5f9e41768b2074130cbe8a4d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 08:08:50 UTC) from {username='oryx', principal='oryx/127.12.201.254@KRBTEST.COM'} at 127.0.0.1:42216
I20250903 08:08:50.242944 21842 ts_tablet_manager.cc:1905] T 6b660cab5f9e41768b2074130cbe8a4d P fa5050fb31d1413a949c35c167d78324: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 08:08:50.245458 21843 tablet_replica.cc:331] T 6b660cab5f9e41768b2074130cbe8a4d P dc83179923a84bfaa56be4f55a7158ef: stopping tablet replica
I20250903 08:08:50.245626 21843 raft_consensus.cc:2241] T 6b660cab5f9e41768b2074130cbe8a4d P dc83179923a84bfaa56be4f55a7158ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 08:08:50.245807 21843 raft_consensus.cc:2270] T 6b660cab5f9e41768b2074130cbe8a4d P dc83179923a84bfaa56be4f55a7158ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 08:08:50.246192 21842 ts_tablet_manager.cc:1918] T 6b660cab5f9e41768b2074130cbe8a4d P fa5050fb31d1413a949c35c167d78324: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20250903 08:08:50.246264 21842 log.cc:1199] T 6b660cab5f9e41768b2074130cbe8a4d P fa5050fb31d1413a949c35c167d78324: Deleting WAL directory at /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/minicluster-data/ts-0/wal/wals/6b660cab5f9e41768b2074130cbe8a4d
I20250903 08:08:50.246543 21842 ts_tablet_manager.cc:1939] T 6b660cab5f9e41768b2074130cbe8a4d P fa5050fb31d1413a949c35c167d78324: Deleting consensus metadata
I20250903 08:08:50.247907 21841 ts_tablet_manager.cc:1905] T 6b660cab5f9e41768b2074130cbe8a4d P 9bf6ef1f80704b49bf1b4354718df875: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 08:08:50.250015 21843 ts_tablet_manager.cc:1905] T 6b660cab5f9e41768b2074130cbe8a4d P dc83179923a84bfaa56be4f55a7158ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 08:08:50.250761 21841 ts_tablet_manager.cc:1918] T 6b660cab5f9e41768b2074130cbe8a4d P 9bf6ef1f80704b49bf1b4354718df875: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20250903 08:08:50.250821 21841 log.cc:1199] T 6b660cab5f9e41768b2074130cbe8a4d P 9bf6ef1f80704b49bf1b4354718df875: Deleting WAL directory at /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/minicluster-data/ts-1/wal/wals/6b660cab5f9e41768b2074130cbe8a4d
I20250903 08:08:50.251382 21841 ts_tablet_manager.cc:1939] T 6b660cab5f9e41768b2074130cbe8a4d P 9bf6ef1f80704b49bf1b4354718df875: Deleting consensus metadata
I20250903 08:08:50.252187 21286 catalog_manager.cc:4928] TS 9bf6ef1f80704b49bf1b4354718df875 (127.12.201.194:46253): tablet 6b660cab5f9e41768b2074130cbe8a4d (table test-table [id=f552d51104834d7e9b3a775fb0468492]) successfully deleted
I20250903 08:08:50.253113 21288 catalog_manager.cc:4928] TS fa5050fb31d1413a949c35c167d78324 (127.12.201.193:34563): tablet 6b660cab5f9e41768b2074130cbe8a4d (table test-table [id=f552d51104834d7e9b3a775fb0468492]) successfully deleted
I20250903 08:08:50.253741 21843 ts_tablet_manager.cc:1918] T 6b660cab5f9e41768b2074130cbe8a4d P dc83179923a84bfaa56be4f55a7158ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20250903 08:08:50.253823 21843 log.cc:1199] T 6b660cab5f9e41768b2074130cbe8a4d P dc83179923a84bfaa56be4f55a7158ef: Deleting WAL directory at /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/minicluster-data/ts-2/wal/wals/6b660cab5f9e41768b2074130cbe8a4d
I20250903 08:08:50.254124 21843 ts_tablet_manager.cc:1939] T 6b660cab5f9e41768b2074130cbe8a4d P dc83179923a84bfaa56be4f55a7158ef: Deleting consensus metadata
I20250903 08:08:50.255012 21285 catalog_manager.cc:4928] TS dc83179923a84bfaa56be4f55a7158ef (127.12.201.195:36249): tablet 6b660cab5f9e41768b2074130cbe8a4d (table test-table [id=f552d51104834d7e9b3a775fb0468492]) successfully deleted
Sep 03 08:08:50 dist-test-slave-27r9 krb5kdc[21253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886930, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 03 08:08:50 dist-test-slave-27r9 krb5kdc[21253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
Sep 03 08:08:50 dist-test-slave-27r9 krb5kdc[21253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
W20250903 08:08:50.307212 21350 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 08:08:50.299471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45688 (local address 127.12.201.254:38257)
0903 08:08:50.299629 (+ 158us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:50.299635 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:50.299887 (+ 252us) server_negotiation.cc:408] Connection header received
0903 08:08:50.300098 (+ 211us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:50.300102 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:50.300157 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:50.300211 (+ 54us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:50.301468 (+ 1257us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:50.302267 (+ 799us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:50.303271 (+ 1004us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:50.303462 (+ 191us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:50.307076 (+ 3614us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45688: BlockingRecv error: recv got EOF from 127.0.0.1:45688 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 03 08:08:50 dist-test-slave-27r9 krb5kdc[21253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886930, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM
I20250903 08:08:50.319864 21350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:50.309422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45702 (local address 127.12.201.254:38257)
0903 08:08:50.309538 (+ 116us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:50.309542 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:50.309927 (+ 385us) server_negotiation.cc:408] Connection header received
0903 08:08:50.310062 (+ 135us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:50.310066 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:50.310119 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:50.310195 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:50.311205 (+ 1010us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:50.311969 (+ 764us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:50.312839 (+ 870us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:50.313025 (+ 186us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:50.316098 (+ 3073us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:50.316120 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:50.316123 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:08:50.316154 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:50.317905 (+ 1751us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:50.318438 (+ 533us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:50.318442 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:50.318444 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:50.318498 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:50.318791 (+ 293us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:50.318795 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:50.318797 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:50.319062 (+ 265us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:50.319176 (+ 114us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:50.319558 (+ 382us) server_negotiation.cc:300] Negotiation successful
0903 08:08:50.319695 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20250903 08:08:50.322839 21301 catalog_manager.cc:2232] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45702:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 08:08:50.323364 21301 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 08:08:50.328900 21440 tablet_service.cc:1468] Processing CreateTablet for tablet 4b1be649adef4db0a6e00091b1a3580d (DEFAULT_TABLE table=test-table [id=9de37eeda7904114b86142bced1f955f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:08:50.329233 21440 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b1be649adef4db0a6e00091b1a3580d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:08:50.331840 21818 tablet_bootstrap.cc:492] T 4b1be649adef4db0a6e00091b1a3580d P fa5050fb31d1413a949c35c167d78324: Bootstrap starting.
I20250903 08:08:50.331947 21582 tablet_service.cc:1468] Processing CreateTablet for tablet 4b1be649adef4db0a6e00091b1a3580d (DEFAULT_TABLE table=test-table [id=9de37eeda7904114b86142bced1f955f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:08:50.332222 21582 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b1be649adef4db0a6e00091b1a3580d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:08:50.332867 21818 tablet_bootstrap.cc:654] T 4b1be649adef4db0a6e00091b1a3580d P fa5050fb31d1413a949c35c167d78324: Neither blocks nor log segments found. Creating new log.
I20250903 08:08:50.334491 21818 tablet_bootstrap.cc:492] T 4b1be649adef4db0a6e00091b1a3580d P fa5050fb31d1413a949c35c167d78324: No bootstrap required, opened a new log
I20250903 08:08:50.334581 21818 ts_tablet_manager.cc:1397] T 4b1be649adef4db0a6e00091b1a3580d P fa5050fb31d1413a949c35c167d78324: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20250903 08:08:50.334836 21724 tablet_service.cc:1468] Processing CreateTablet for tablet 4b1be649adef4db0a6e00091b1a3580d (DEFAULT_TABLE table=test-table [id=9de37eeda7904114b86142bced1f955f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:08:50.335064 21818 raft_consensus.cc:357] T 4b1be649adef4db0a6e00091b1a3580d P fa5050fb31d1413a949c35c167d78324 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5050fb31d1413a949c35c167d78324" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 34563 } } peers { permanent_uuid: "9bf6ef1f80704b49bf1b4354718df875" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 46253 } } peers { permanent_uuid: "dc83179923a84bfaa56be4f55a7158ef" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 36249 } }
I20250903 08:08:50.335175 21818 raft_consensus.cc:383] T 4b1be649adef4db0a6e00091b1a3580d P fa5050fb31d1413a949c35c167d78324 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:08:50.335211 21818 raft_consensus.cc:738] T 4b1be649adef4db0a6e00091b1a3580d P fa5050fb31d1413a949c35c167d78324 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa5050fb31d1413a949c35c167d78324, State: Initialized, Role: FOLLOWER
I20250903 08:08:50.335359 21818 consensus_queue.cc:260] T 4b1be649adef4db0a6e00091b1a3580d P fa5050fb31d1413a949c35c167d78324 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5050fb31d1413a949c35c167d78324" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 34563 } } peers { permanent_uuid: "9bf6ef1f80704b49bf1b4354718df875" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 46253 } } peers { permanent_uuid: "dc83179923a84bfaa56be4f55a7158ef" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 36249 } }
I20250903 08:08:50.335551 21724 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b1be649adef4db0a6e00091b1a3580d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:08:50.335651 21818 ts_tablet_manager.cc:1428] T 4b1be649adef4db0a6e00091b1a3580d P fa5050fb31d1413a949c35c167d78324: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250903 08:08:50.335903 21820 tablet_bootstrap.cc:492] T 4b1be649adef4db0a6e00091b1a3580d P 9bf6ef1f80704b49bf1b4354718df875: Bootstrap starting.
I20250903 08:08:50.336930 21820 tablet_bootstrap.cc:654] T 4b1be649adef4db0a6e00091b1a3580d P 9bf6ef1f80704b49bf1b4354718df875: Neither blocks nor log segments found. Creating new log.
I20250903 08:08:50.337858 21819 tablet_bootstrap.cc:492] T 4b1be649adef4db0a6e00091b1a3580d P dc83179923a84bfaa56be4f55a7158ef: Bootstrap starting.
I20250903 08:08:50.338805 21819 tablet_bootstrap.cc:654] T 4b1be649adef4db0a6e00091b1a3580d P dc83179923a84bfaa56be4f55a7158ef: Neither blocks nor log segments found. Creating new log.
I20250903 08:08:50.339169 21820 tablet_bootstrap.cc:492] T 4b1be649adef4db0a6e00091b1a3580d P 9bf6ef1f80704b49bf1b4354718df875: No bootstrap required, opened a new log
I20250903 08:08:50.339255 21820 ts_tablet_manager.cc:1397] T 4b1be649adef4db0a6e00091b1a3580d P 9bf6ef1f80704b49bf1b4354718df875: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20250903 08:08:50.339761 21820 raft_consensus.cc:357] T 4b1be649adef4db0a6e00091b1a3580d P 9bf6ef1f80704b49bf1b4354718df875 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5050fb31d1413a949c35c167d78324" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 34563 } } peers { permanent_uuid: "9bf6ef1f80704b49bf1b4354718df875" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 46253 } } peers { permanent_uuid: "dc83179923a84bfaa56be4f55a7158ef" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 36249 } }
I20250903 08:08:50.339881 21820 raft_consensus.cc:383] T 4b1be649adef4db0a6e00091b1a3580d P 9bf6ef1f80704b49bf1b4354718df875 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:08:50.339917 21820 raft_consensus.cc:738] T 4b1be649adef4db0a6e00091b1a3580d P 9bf6ef1f80704b49bf1b4354718df875 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bf6ef1f80704b49bf1b4354718df875, State: Initialized, Role: FOLLOWER
I20250903 08:08:50.340057 21820 consensus_queue.cc:260] T 4b1be649adef4db0a6e00091b1a3580d P 9bf6ef1f80704b49bf1b4354718df875 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5050fb31d1413a949c35c167d78324" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 34563 } } peers { permanent_uuid: "9bf6ef1f80704b49bf1b4354718df875" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 46253 } } peers { permanent_uuid: "dc83179923a84bfaa56be4f55a7158ef" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 36249 } }
I20250903 08:08:50.340351 21820 ts_tablet_manager.cc:1428] T 4b1be649adef4db0a6e00091b1a3580d P 9bf6ef1f80704b49bf1b4354718df875: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250903 08:08:50.342414 21819 tablet_bootstrap.cc:492] T 4b1be649adef4db0a6e00091b1a3580d P dc83179923a84bfaa56be4f55a7158ef: No bootstrap required, opened a new log
I20250903 08:08:50.342511 21819 ts_tablet_manager.cc:1397] T 4b1be649adef4db0a6e00091b1a3580d P dc83179923a84bfaa56be4f55a7158ef: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20250903 08:08:50.343026 21819 raft_consensus.cc:357] T 4b1be649adef4db0a6e00091b1a3580d P dc83179923a84bfaa56be4f55a7158ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5050fb31d1413a949c35c167d78324" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 34563 } } peers { permanent_uuid: "9bf6ef1f80704b49bf1b4354718df875" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 46253 } } peers { permanent_uuid: "dc83179923a84bfaa56be4f55a7158ef" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 36249 } }
I20250903 08:08:50.343139 21819 raft_consensus.cc:383] T 4b1be649adef4db0a6e00091b1a3580d P dc83179923a84bfaa56be4f55a7158ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:08:50.343171 21819 raft_consensus.cc:738] T 4b1be649adef4db0a6e00091b1a3580d P dc83179923a84bfaa56be4f55a7158ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc83179923a84bfaa56be4f55a7158ef, State: Initialized, Role: FOLLOWER
I20250903 08:08:50.343322 21819 consensus_queue.cc:260] T 4b1be649adef4db0a6e00091b1a3580d P dc83179923a84bfaa56be4f55a7158ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5050fb31d1413a949c35c167d78324" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 34563 } } peers { permanent_uuid: "9bf6ef1f80704b49bf1b4354718df875" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 46253 } } peers { permanent_uuid: "dc83179923a84bfaa56be4f55a7158ef" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 36249 } }
I20250903 08:08:50.343607 21819 ts_tablet_manager.cc:1428] T 4b1be649adef4db0a6e00091b1a3580d P dc83179923a84bfaa56be4f55a7158ef: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250903 08:08:50.537120 21826 raft_consensus.cc:491] T 4b1be649adef4db0a6e00091b1a3580d P 9bf6ef1f80704b49bf1b4354718df875 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 08:08:50.537262 21826 raft_consensus.cc:513] T 4b1be649adef4db0a6e00091b1a3580d P 9bf6ef1f80704b49bf1b4354718df875 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5050fb31d1413a949c35c167d78324" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 34563 } } peers { permanent_uuid: "9bf6ef1f80704b49bf1b4354718df875" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 46253 } } peers { permanent_uuid: "dc83179923a84bfaa56be4f55a7158ef" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 36249 } }
I20250903 08:08:50.537721 21826 leader_election.cc:290] T 4b1be649adef4db0a6e00091b1a3580d P 9bf6ef1f80704b49bf1b4354718df875 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa5050fb31d1413a949c35c167d78324 (127.12.201.193:34563), dc83179923a84bfaa56be4f55a7158ef (127.12.201.195:36249)
I20250903 08:08:50.538195 21744 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4b1be649adef4db0a6e00091b1a3580d" candidate_uuid: "9bf6ef1f80704b49bf1b4354718df875" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc83179923a84bfaa56be4f55a7158ef" is_pre_election: true
I20250903 08:08:50.538196 21446 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4b1be649adef4db0a6e00091b1a3580d" candidate_uuid: "9bf6ef1f80704b49bf1b4354718df875" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa5050fb31d1413a949c35c167d78324" is_pre_election: true
I20250903 08:08:50.538353 21744 raft_consensus.cc:2466] T 4b1be649adef4db0a6e00091b1a3580d P dc83179923a84bfaa56be4f55a7158ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bf6ef1f80704b49bf1b4354718df875 in term 0.
I20250903 08:08:50.538380 21446 raft_consensus.cc:2466] T 4b1be649adef4db0a6e00091b1a3580d P fa5050fb31d1413a949c35c167d78324 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bf6ef1f80704b49bf1b4354718df875 in term 0.
I20250903 08:08:50.538663 21533 leader_election.cc:304] T 4b1be649adef4db0a6e00091b1a3580d P 9bf6ef1f80704b49bf1b4354718df875 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bf6ef1f80704b49bf1b4354718df875, dc83179923a84bfaa56be4f55a7158ef; no voters:
I20250903 08:08:50.538853 21826 raft_consensus.cc:2802] T 4b1be649adef4db0a6e00091b1a3580d P 9bf6ef1f80704b49bf1b4354718df875 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 08:08:50.538910 21826 raft_consensus.cc:491] T 4b1be649adef4db0a6e00091b1a3580d P 9bf6ef1f80704b49bf1b4354718df875 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 08:08:50.538965 21826 raft_consensus.cc:3058] T 4b1be649adef4db0a6e00091b1a3580d P 9bf6ef1f80704b49bf1b4354718df875 [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:08:50.539891 21826 raft_consensus.cc:513] T 4b1be649adef4db0a6e00091b1a3580d P 9bf6ef1f80704b49bf1b4354718df875 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5050fb31d1413a949c35c167d78324" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 34563 } } peers { permanent_uuid: "9bf6ef1f80704b49bf1b4354718df875" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 46253 } } peers { permanent_uuid: "dc83179923a84bfaa56be4f55a7158ef" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 36249 } }
I20250903 08:08:50.540202 21826 leader_election.cc:290] T 4b1be649adef4db0a6e00091b1a3580d P 9bf6ef1f80704b49bf1b4354718df875 [CANDIDATE]: Term 1 election: Requested vote from peers fa5050fb31d1413a949c35c167d78324 (127.12.201.193:34563), dc83179923a84bfaa56be4f55a7158ef (127.12.201.195:36249)
I20250903 08:08:50.540567 21744 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4b1be649adef4db0a6e00091b1a3580d" candidate_uuid: "9bf6ef1f80704b49bf1b4354718df875" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc83179923a84bfaa56be4f55a7158ef"
I20250903 08:08:50.540567 21446 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4b1be649adef4db0a6e00091b1a3580d" candidate_uuid: "9bf6ef1f80704b49bf1b4354718df875" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa5050fb31d1413a949c35c167d78324"
I20250903 08:08:50.540664 21446 raft_consensus.cc:3058] T 4b1be649adef4db0a6e00091b1a3580d P fa5050fb31d1413a949c35c167d78324 [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:08:50.540675 21744 raft_consensus.cc:3058] T 4b1be649adef4db0a6e00091b1a3580d P dc83179923a84bfaa56be4f55a7158ef [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:08:50.541486 21744 raft_consensus.cc:2466] T 4b1be649adef4db0a6e00091b1a3580d P dc83179923a84bfaa56be4f55a7158ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bf6ef1f80704b49bf1b4354718df875 in term 1.
I20250903 08:08:50.541529 21446 raft_consensus.cc:2466] T 4b1be649adef4db0a6e00091b1a3580d P fa5050fb31d1413a949c35c167d78324 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bf6ef1f80704b49bf1b4354718df875 in term 1.
I20250903 08:08:50.541788 21533 leader_election.cc:304] T 4b1be649adef4db0a6e00091b1a3580d P 9bf6ef1f80704b49bf1b4354718df875 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bf6ef1f80704b49bf1b4354718df875, dc83179923a84bfaa56be4f55a7158ef; no voters:
I20250903 08:08:50.541967 21826 raft_consensus.cc:2802] T 4b1be649adef4db0a6e00091b1a3580d P 9bf6ef1f80704b49bf1b4354718df875 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 08:08:50.542042 21826 raft_consensus.cc:695] T 4b1be649adef4db0a6e00091b1a3580d P 9bf6ef1f80704b49bf1b4354718df875 [term 1 LEADER]: Becoming Leader. State: Replica: 9bf6ef1f80704b49bf1b4354718df875, State: Running, Role: LEADER
I20250903 08:08:50.542181 21826 consensus_queue.cc:237] T 4b1be649adef4db0a6e00091b1a3580d P 9bf6ef1f80704b49bf1b4354718df875 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5050fb31d1413a949c35c167d78324" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 34563 } } peers { permanent_uuid: "9bf6ef1f80704b49bf1b4354718df875" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 46253 } } peers { permanent_uuid: "dc83179923a84bfaa56be4f55a7158ef" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 36249 } }
I20250903 08:08:50.543809 21301 catalog_manager.cc:5582] T 4b1be649adef4db0a6e00091b1a3580d P 9bf6ef1f80704b49bf1b4354718df875 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9bf6ef1f80704b49bf1b4354718df875 (127.12.201.194). New cstate: current_term: 1 leader_uuid: "9bf6ef1f80704b49bf1b4354718df875" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5050fb31d1413a949c35c167d78324" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 34563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9bf6ef1f80704b49bf1b4354718df875" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 46253 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc83179923a84bfaa56be4f55a7158ef" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 36249 } health_report { overall_health: UNKNOWN } } }
I20250903 08:08:50.582373 21814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:50.580028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49352 (local address 127.12.201.194:46253)
0903 08:08:50.580148 (+ 120us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:50.580152 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:50.580165 (+ 13us) server_negotiation.cc:408] Connection header received
0903 08:08:50.580209 (+ 44us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:50.580212 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:50.580260 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:50.580329 (+ 69us) server_negotiation.cc:227] Negotiated authn=TOKEN
0903 08:08:50.580644 (+ 315us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:50.581100 (+ 456us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:50.581732 (+ 632us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:50.581860 (+ 128us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:50.581899 (+ 39us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0903 08:08:50.581963 (+ 64us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0903 08:08:50.582024 (+ 61us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:50.582215 (+ 191us) server_negotiation.cc:300] Negotiation successful
0903 08:08:50.582264 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20250903 08:08:50.584452 21744 raft_consensus.cc:1273] T 4b1be649adef4db0a6e00091b1a3580d P dc83179923a84bfaa56be4f55a7158ef [term 1 FOLLOWER]: Refusing update from remote peer 9bf6ef1f80704b49bf1b4354718df875: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 08:08:50.584554 21446 raft_consensus.cc:1273] T 4b1be649adef4db0a6e00091b1a3580d P fa5050fb31d1413a949c35c167d78324 [term 1 FOLLOWER]: Refusing update from remote peer 9bf6ef1f80704b49bf1b4354718df875: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 08:08:50.584893 21826 consensus_queue.cc:1035] T 4b1be649adef4db0a6e00091b1a3580d P 9bf6ef1f80704b49bf1b4354718df875 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc83179923a84bfaa56be4f55a7158ef" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 36249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 08:08:50.585057 21849 consensus_queue.cc:1035] T 4b1be649adef4db0a6e00091b1a3580d P 9bf6ef1f80704b49bf1b4354718df875 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa5050fb31d1413a949c35c167d78324" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 08:08:50.588181 21834 mvcc.cc:204] Tried to move back new op lower bound from 7196208867669790720 to 7196208867502161920. Current Snapshot: MvccSnapshot[applied={T|T < 7196208867669790720}]
I20250903 08:08:50.592061 21301 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45702:
table { table_name: "test-table" } modify_external_catalogs: true
I20250903 08:08:50.592178 21301 catalog_manager.cc:2730] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45702:
table { table_name: "test-table" } modify_external_catalogs: true
I20250903 08:08:50.594797 21301 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3aa107cc15d641bca7f21dd4251fdf15: Sending DeleteTablet for 3 replicas of tablet 4b1be649adef4db0a6e00091b1a3580d
I20250903 08:08:50.595374 21440 tablet_service.cc:1515] Processing DeleteTablet for tablet 4b1be649adef4db0a6e00091b1a3580d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 08:08:50 UTC) from {username='oryx', principal='oryx/127.12.201.254@KRBTEST.COM'} at 127.0.0.1:54102
I20250903 08:08:50.595460 21724 tablet_service.cc:1515] Processing DeleteTablet for tablet 4b1be649adef4db0a6e00091b1a3580d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 08:08:50 UTC) from {username='oryx', principal='oryx/127.12.201.254@KRBTEST.COM'} at 127.0.0.1:42216
I20250903 08:08:50.595562 21842 tablet_replica.cc:331] T 4b1be649adef4db0a6e00091b1a3580d P fa5050fb31d1413a949c35c167d78324: stopping tablet replica
I20250903 08:08:50.595649 21843 tablet_replica.cc:331] T 4b1be649adef4db0a6e00091b1a3580d P dc83179923a84bfaa56be4f55a7158ef: stopping tablet replica
I20250903 08:08:50.595677 21842 raft_consensus.cc:2241] T 4b1be649adef4db0a6e00091b1a3580d P fa5050fb31d1413a949c35c167d78324 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 08:08:50.595683 21582 tablet_service.cc:1515] Processing DeleteTablet for tablet 4b1be649adef4db0a6e00091b1a3580d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 08:08:50 UTC) from {username='oryx', principal='oryx/127.12.201.254@KRBTEST.COM'} at 127.0.0.1:49330
I20250903 08:08:50.595764 21843 raft_consensus.cc:2241] T 4b1be649adef4db0a6e00091b1a3580d P dc83179923a84bfaa56be4f55a7158ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 08:08:50.595799 21842 raft_consensus.cc:2270] T 4b1be649adef4db0a6e00091b1a3580d P fa5050fb31d1413a949c35c167d78324 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 08:08:50.595827 21841 tablet_replica.cc:331] T 4b1be649adef4db0a6e00091b1a3580d P 9bf6ef1f80704b49bf1b4354718df875: stopping tablet replica
I20250903 08:08:50.595888 21843 raft_consensus.cc:2270] T 4b1be649adef4db0a6e00091b1a3580d P dc83179923a84bfaa56be4f55a7158ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 08:08:50.595925 21841 raft_consensus.cc:2241] T 4b1be649adef4db0a6e00091b1a3580d P 9bf6ef1f80704b49bf1b4354718df875 [term 1 LEADER]: Raft consensus shutting down.
I20250903 08:08:50.596099 21841 raft_consensus.cc:2270] T 4b1be649adef4db0a6e00091b1a3580d P 9bf6ef1f80704b49bf1b4354718df875 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 08:08:50.596494 21582 tablet_service.cc:1515] Processing DeleteTablet for tablet 4b1be649adef4db0a6e00091b1a3580d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 08:08:50 UTC) from {username='oryx', principal='oryx/127.12.201.254@KRBTEST.COM'} at 127.0.0.1:49330
I20250903 08:08:50.596634 21843 ts_tablet_manager.cc:1905] T 4b1be649adef4db0a6e00091b1a3580d P dc83179923a84bfaa56be4f55a7158ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 08:08:50.596750 21842 ts_tablet_manager.cc:1905] T 4b1be649adef4db0a6e00091b1a3580d P fa5050fb31d1413a949c35c167d78324: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 08:08:50.597612 21300 catalog_manager.cc:2232] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45702:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 08:08:50.597728 21841 ts_tablet_manager.cc:1905] T 4b1be649adef4db0a6e00091b1a3580d P 9bf6ef1f80704b49bf1b4354718df875: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250903 08:08:50.598068 21300 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 08:08:50.598917 21843 ts_tablet_manager.cc:1918] T 4b1be649adef4db0a6e00091b1a3580d P dc83179923a84bfaa56be4f55a7158ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250903 08:08:50.599020 21843 log.cc:1199] T 4b1be649adef4db0a6e00091b1a3580d P dc83179923a84bfaa56be4f55a7158ef: Deleting WAL directory at /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/minicluster-data/ts-2/wal/wals/4b1be649adef4db0a6e00091b1a3580d
I20250903 08:08:50.599136 21842 ts_tablet_manager.cc:1918] T 4b1be649adef4db0a6e00091b1a3580d P fa5050fb31d1413a949c35c167d78324: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250903 08:08:50.599203 21842 log.cc:1199] T 4b1be649adef4db0a6e00091b1a3580d P fa5050fb31d1413a949c35c167d78324: Deleting WAL directory at /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/minicluster-data/ts-0/wal/wals/4b1be649adef4db0a6e00091b1a3580d
I20250903 08:08:50.599270 21843 ts_tablet_manager.cc:1939] T 4b1be649adef4db0a6e00091b1a3580d P dc83179923a84bfaa56be4f55a7158ef: Deleting consensus metadata
I20250903 08:08:50.599452 21842 ts_tablet_manager.cc:1939] T 4b1be649adef4db0a6e00091b1a3580d P fa5050fb31d1413a949c35c167d78324: Deleting consensus metadata
I20250903 08:08:50.599992 21841 ts_tablet_manager.cc:1918] T 4b1be649adef4db0a6e00091b1a3580d P 9bf6ef1f80704b49bf1b4354718df875: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250903 08:08:50.600069 21841 log.cc:1199] T 4b1be649adef4db0a6e00091b1a3580d P 9bf6ef1f80704b49bf1b4354718df875: Deleting WAL directory at /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/minicluster-data/ts-1/wal/wals/4b1be649adef4db0a6e00091b1a3580d
I20250903 08:08:50.600072 21288 catalog_manager.cc:4928] TS fa5050fb31d1413a949c35c167d78324 (127.12.201.193:34563): tablet 4b1be649adef4db0a6e00091b1a3580d (table test-table [id=9de37eeda7904114b86142bced1f955f]) successfully deleted
I20250903 08:08:50.600302 21841 ts_tablet_manager.cc:1939] T 4b1be649adef4db0a6e00091b1a3580d P 9bf6ef1f80704b49bf1b4354718df875: Deleting consensus metadata
I20250903 08:08:50.601114 21286 catalog_manager.cc:4928] TS 9bf6ef1f80704b49bf1b4354718df875 (127.12.201.194:46253): tablet 4b1be649adef4db0a6e00091b1a3580d (table test-table [id=9de37eeda7904114b86142bced1f955f]) successfully deleted
W20250903 08:08:50.601359 21286 catalog_manager.cc:4895] TS 9bf6ef1f80704b49bf1b4354718df875 (127.12.201.194:46253): delete failed for tablet 4b1be649adef4db0a6e00091b1a3580d because the tablet was not found. No further retry: Not found: Tablet not found: 4b1be649adef4db0a6e00091b1a3580d
I20250903 08:08:50.601723 21285 catalog_manager.cc:4928] TS dc83179923a84bfaa56be4f55a7158ef (127.12.201.195:36249): tablet 4b1be649adef4db0a6e00091b1a3580d (table test-table [id=9de37eeda7904114b86142bced1f955f]) successfully deleted
I20250903 08:08:50.603349 21440 tablet_service.cc:1468] Processing CreateTablet for tablet a04dbc4a62c040f5bd7feae8d00b725d (DEFAULT_TABLE table=test-table [id=3839c6f66f194c2b992323bf68b2ffce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:08:50.603368 21582 tablet_service.cc:1468] Processing CreateTablet for tablet a04dbc4a62c040f5bd7feae8d00b725d (DEFAULT_TABLE table=test-table [id=3839c6f66f194c2b992323bf68b2ffce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:08:50.603531 21724 tablet_service.cc:1468] Processing CreateTablet for tablet a04dbc4a62c040f5bd7feae8d00b725d (DEFAULT_TABLE table=test-table [id=3839c6f66f194c2b992323bf68b2ffce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:08:50.603600 21440 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a04dbc4a62c040f5bd7feae8d00b725d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:08:50.603610 21582 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a04dbc4a62c040f5bd7feae8d00b725d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:08:50.603773 21724 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a04dbc4a62c040f5bd7feae8d00b725d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:08:50.605306 21819 tablet_bootstrap.cc:492] T a04dbc4a62c040f5bd7feae8d00b725d P dc83179923a84bfaa56be4f55a7158ef: Bootstrap starting.
I20250903 08:08:50.605659 21818 tablet_bootstrap.cc:492] T a04dbc4a62c040f5bd7feae8d00b725d P fa5050fb31d1413a949c35c167d78324: Bootstrap starting.
I20250903 08:08:50.606087 21819 tablet_bootstrap.cc:654] T a04dbc4a62c040f5bd7feae8d00b725d P dc83179923a84bfaa56be4f55a7158ef: Neither blocks nor log segments found. Creating new log.
I20250903 08:08:50.606352 21818 tablet_bootstrap.cc:654] T a04dbc4a62c040f5bd7feae8d00b725d P fa5050fb31d1413a949c35c167d78324: Neither blocks nor log segments found. Creating new log.
I20250903 08:08:50.606822 21820 tablet_bootstrap.cc:492] T a04dbc4a62c040f5bd7feae8d00b725d P 9bf6ef1f80704b49bf1b4354718df875: Bootstrap starting.
I20250903 08:08:50.607074 21819 tablet_bootstrap.cc:492] T a04dbc4a62c040f5bd7feae8d00b725d P dc83179923a84bfaa56be4f55a7158ef: No bootstrap required, opened a new log
I20250903 08:08:50.607133 21819 ts_tablet_manager.cc:1397] T a04dbc4a62c040f5bd7feae8d00b725d P dc83179923a84bfaa56be4f55a7158ef: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250903 08:08:50.607426 21819 raft_consensus.cc:357] T a04dbc4a62c040f5bd7feae8d00b725d P dc83179923a84bfaa56be4f55a7158ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5050fb31d1413a949c35c167d78324" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 34563 } } peers { permanent_uuid: "9bf6ef1f80704b49bf1b4354718df875" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 46253 } } peers { permanent_uuid: "dc83179923a84bfaa56be4f55a7158ef" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 36249 } }
I20250903 08:08:50.607498 21819 raft_consensus.cc:383] T a04dbc4a62c040f5bd7feae8d00b725d P dc83179923a84bfaa56be4f55a7158ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:08:50.607518 21819 raft_consensus.cc:738] T a04dbc4a62c040f5bd7feae8d00b725d P dc83179923a84bfaa56be4f55a7158ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc83179923a84bfaa56be4f55a7158ef, State: Initialized, Role: FOLLOWER
I20250903 08:08:50.607535 21820 tablet_bootstrap.cc:654] T a04dbc4a62c040f5bd7feae8d00b725d P 9bf6ef1f80704b49bf1b4354718df875: Neither blocks nor log segments found. Creating new log.
I20250903 08:08:50.607622 21819 consensus_queue.cc:260] T a04dbc4a62c040f5bd7feae8d00b725d P dc83179923a84bfaa56be4f55a7158ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5050fb31d1413a949c35c167d78324" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 34563 } } peers { permanent_uuid: "9bf6ef1f80704b49bf1b4354718df875" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 46253 } } peers { permanent_uuid: "dc83179923a84bfaa56be4f55a7158ef" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 36249 } }
I20250903 08:08:50.607836 21819 ts_tablet_manager.cc:1428] T a04dbc4a62c040f5bd7feae8d00b725d P dc83179923a84bfaa56be4f55a7158ef: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250903 08:08:50.608287 21818 tablet_bootstrap.cc:492] T a04dbc4a62c040f5bd7feae8d00b725d P fa5050fb31d1413a949c35c167d78324: No bootstrap required, opened a new log
I20250903 08:08:50.608379 21818 ts_tablet_manager.cc:1397] T a04dbc4a62c040f5bd7feae8d00b725d P fa5050fb31d1413a949c35c167d78324: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20250903 08:08:50.608559 21820 tablet_bootstrap.cc:492] T a04dbc4a62c040f5bd7feae8d00b725d P 9bf6ef1f80704b49bf1b4354718df875: No bootstrap required, opened a new log
I20250903 08:08:50.608639 21820 ts_tablet_manager.cc:1397] T a04dbc4a62c040f5bd7feae8d00b725d P 9bf6ef1f80704b49bf1b4354718df875: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250903 08:08:50.608776 21818 raft_consensus.cc:357] T a04dbc4a62c040f5bd7feae8d00b725d P fa5050fb31d1413a949c35c167d78324 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5050fb31d1413a949c35c167d78324" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 34563 } } peers { permanent_uuid: "9bf6ef1f80704b49bf1b4354718df875" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 46253 } } peers { permanent_uuid: "dc83179923a84bfaa56be4f55a7158ef" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 36249 } }
I20250903 08:08:50.608868 21818 raft_consensus.cc:383] T a04dbc4a62c040f5bd7feae8d00b725d P fa5050fb31d1413a949c35c167d78324 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:08:50.608914 21818 raft_consensus.cc:738] T a04dbc4a62c040f5bd7feae8d00b725d P fa5050fb31d1413a949c35c167d78324 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa5050fb31d1413a949c35c167d78324, State: Initialized, Role: FOLLOWER
I20250903 08:08:50.609030 21820 raft_consensus.cc:357] T a04dbc4a62c040f5bd7feae8d00b725d P 9bf6ef1f80704b49bf1b4354718df875 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5050fb31d1413a949c35c167d78324" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 34563 } } peers { permanent_uuid: "9bf6ef1f80704b49bf1b4354718df875" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 46253 } } peers { permanent_uuid: "dc83179923a84bfaa56be4f55a7158ef" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 36249 } }
I20250903 08:08:50.609042 21818 consensus_queue.cc:260] T a04dbc4a62c040f5bd7feae8d00b725d P fa5050fb31d1413a949c35c167d78324 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5050fb31d1413a949c35c167d78324" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 34563 } } peers { permanent_uuid: "9bf6ef1f80704b49bf1b4354718df875" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 46253 } } peers { permanent_uuid: "dc83179923a84bfaa56be4f55a7158ef" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 36249 } }
I20250903 08:08:50.609117 21820 raft_consensus.cc:383] T a04dbc4a62c040f5bd7feae8d00b725d P 9bf6ef1f80704b49bf1b4354718df875 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:08:50.609138 21820 raft_consensus.cc:738] T a04dbc4a62c040f5bd7feae8d00b725d P 9bf6ef1f80704b49bf1b4354718df875 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bf6ef1f80704b49bf1b4354718df875, State: Initialized, Role: FOLLOWER
I20250903 08:08:50.609266 21820 consensus_queue.cc:260] T a04dbc4a62c040f5bd7feae8d00b725d P 9bf6ef1f80704b49bf1b4354718df875 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5050fb31d1413a949c35c167d78324" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 34563 } } peers { permanent_uuid: "9bf6ef1f80704b49bf1b4354718df875" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 46253 } } peers { permanent_uuid: "dc83179923a84bfaa56be4f55a7158ef" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 36249 } }
I20250903 08:08:50.609302 21818 ts_tablet_manager.cc:1428] T a04dbc4a62c040f5bd7feae8d00b725d P fa5050fb31d1413a949c35c167d78324: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250903 08:08:50.609508 21820 ts_tablet_manager.cc:1428] T a04dbc4a62c040f5bd7feae8d00b725d P 9bf6ef1f80704b49bf1b4354718df875: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250903 08:08:50.687201 21825 raft_consensus.cc:491] T a04dbc4a62c040f5bd7feae8d00b725d P fa5050fb31d1413a949c35c167d78324 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 08:08:50.687367 21825 raft_consensus.cc:513] T a04dbc4a62c040f5bd7feae8d00b725d P fa5050fb31d1413a949c35c167d78324 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5050fb31d1413a949c35c167d78324" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 34563 } } peers { permanent_uuid: "9bf6ef1f80704b49bf1b4354718df875" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 46253 } } peers { permanent_uuid: "dc83179923a84bfaa56be4f55a7158ef" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 36249 } }
I20250903 08:08:50.687847 21825 leader_election.cc:290] T a04dbc4a62c040f5bd7feae8d00b725d P fa5050fb31d1413a949c35c167d78324 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bf6ef1f80704b49bf1b4354718df875 (127.12.201.194:46253), dc83179923a84bfaa56be4f55a7158ef (127.12.201.195:36249)
I20250903 08:08:50.688341 21602 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a04dbc4a62c040f5bd7feae8d00b725d" candidate_uuid: "fa5050fb31d1413a949c35c167d78324" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bf6ef1f80704b49bf1b4354718df875" is_pre_election: true
I20250903 08:08:50.688357 21744 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a04dbc4a62c040f5bd7feae8d00b725d" candidate_uuid: "fa5050fb31d1413a949c35c167d78324" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc83179923a84bfaa56be4f55a7158ef" is_pre_election: true
I20250903 08:08:50.688488 21602 raft_consensus.cc:2466] T a04dbc4a62c040f5bd7feae8d00b725d P 9bf6ef1f80704b49bf1b4354718df875 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa5050fb31d1413a949c35c167d78324 in term 0.
I20250903 08:08:50.688516 21744 raft_consensus.cc:2466] T a04dbc4a62c040f5bd7feae8d00b725d P dc83179923a84bfaa56be4f55a7158ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa5050fb31d1413a949c35c167d78324 in term 0.
I20250903 08:08:50.688828 21392 leader_election.cc:304] T a04dbc4a62c040f5bd7feae8d00b725d P fa5050fb31d1413a949c35c167d78324 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bf6ef1f80704b49bf1b4354718df875, fa5050fb31d1413a949c35c167d78324; no voters:
I20250903 08:08:50.689016 21825 raft_consensus.cc:2802] T a04dbc4a62c040f5bd7feae8d00b725d P fa5050fb31d1413a949c35c167d78324 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 08:08:50.689061 21825 raft_consensus.cc:491] T a04dbc4a62c040f5bd7feae8d00b725d P fa5050fb31d1413a949c35c167d78324 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 08:08:50.689095 21825 raft_consensus.cc:3058] T a04dbc4a62c040f5bd7feae8d00b725d P fa5050fb31d1413a949c35c167d78324 [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:08:50.690243 21825 raft_consensus.cc:513] T a04dbc4a62c040f5bd7feae8d00b725d P fa5050fb31d1413a949c35c167d78324 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5050fb31d1413a949c35c167d78324" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 34563 } } peers { permanent_uuid: "9bf6ef1f80704b49bf1b4354718df875" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 46253 } } peers { permanent_uuid: "dc83179923a84bfaa56be4f55a7158ef" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 36249 } }
I20250903 08:08:50.690637 21825 leader_election.cc:290] T a04dbc4a62c040f5bd7feae8d00b725d P fa5050fb31d1413a949c35c167d78324 [CANDIDATE]: Term 1 election: Requested vote from peers 9bf6ef1f80704b49bf1b4354718df875 (127.12.201.194:46253), dc83179923a84bfaa56be4f55a7158ef (127.12.201.195:36249)
I20250903 08:08:50.691030 21602 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a04dbc4a62c040f5bd7feae8d00b725d" candidate_uuid: "fa5050fb31d1413a949c35c167d78324" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bf6ef1f80704b49bf1b4354718df875"
I20250903 08:08:50.691048 21744 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a04dbc4a62c040f5bd7feae8d00b725d" candidate_uuid: "fa5050fb31d1413a949c35c167d78324" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc83179923a84bfaa56be4f55a7158ef"
I20250903 08:08:50.691159 21744 raft_consensus.cc:3058] T a04dbc4a62c040f5bd7feae8d00b725d P dc83179923a84bfaa56be4f55a7158ef [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:08:50.691169 21602 raft_consensus.cc:3058] T a04dbc4a62c040f5bd7feae8d00b725d P 9bf6ef1f80704b49bf1b4354718df875 [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:08:50.692065 21602 raft_consensus.cc:2466] T a04dbc4a62c040f5bd7feae8d00b725d P 9bf6ef1f80704b49bf1b4354718df875 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa5050fb31d1413a949c35c167d78324 in term 1.
I20250903 08:08:50.692179 21744 raft_consensus.cc:2466] T a04dbc4a62c040f5bd7feae8d00b725d P dc83179923a84bfaa56be4f55a7158ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa5050fb31d1413a949c35c167d78324 in term 1.
I20250903 08:08:50.692437 21392 leader_election.cc:304] T a04dbc4a62c040f5bd7feae8d00b725d P fa5050fb31d1413a949c35c167d78324 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bf6ef1f80704b49bf1b4354718df875, fa5050fb31d1413a949c35c167d78324; no voters:
I20250903 08:08:50.692646 21825 raft_consensus.cc:2802] T a04dbc4a62c040f5bd7feae8d00b725d P fa5050fb31d1413a949c35c167d78324 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 08:08:50.692828 21825 raft_consensus.cc:695] T a04dbc4a62c040f5bd7feae8d00b725d P fa5050fb31d1413a949c35c167d78324 [term 1 LEADER]: Becoming Leader. State: Replica: fa5050fb31d1413a949c35c167d78324, State: Running, Role: LEADER
I20250903 08:08:50.692975 21825 consensus_queue.cc:237] T a04dbc4a62c040f5bd7feae8d00b725d P fa5050fb31d1413a949c35c167d78324 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5050fb31d1413a949c35c167d78324" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 34563 } } peers { permanent_uuid: "9bf6ef1f80704b49bf1b4354718df875" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 46253 } } peers { permanent_uuid: "dc83179923a84bfaa56be4f55a7158ef" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 36249 } }
I20250903 08:08:50.694756 21301 catalog_manager.cc:5582] T a04dbc4a62c040f5bd7feae8d00b725d P fa5050fb31d1413a949c35c167d78324 reported cstate change: term changed from 0 to 1, leader changed from <none> to fa5050fb31d1413a949c35c167d78324 (127.12.201.193). New cstate: current_term: 1 leader_uuid: "fa5050fb31d1413a949c35c167d78324" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5050fb31d1413a949c35c167d78324" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 34563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9bf6ef1f80704b49bf1b4354718df875" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 46253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc83179923a84bfaa56be4f55a7158ef" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 36249 } health_report { overall_health: UNKNOWN } } }
I20250903 08:08:50.719745 21905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:50.717300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54114 (local address 127.12.201.193:34563)
0903 08:08:50.717525 (+ 225us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:50.717529 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:50.717541 (+ 12us) server_negotiation.cc:408] Connection header received
0903 08:08:50.717580 (+ 39us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:50.717583 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:50.717628 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:50.717701 (+ 73us) server_negotiation.cc:227] Negotiated authn=TOKEN
0903 08:08:50.717973 (+ 272us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:50.718466 (+ 493us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:50.719152 (+ 686us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:50.719336 (+ 184us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:50.719391 (+ 55us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0903 08:08:50.719460 (+ 69us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0903 08:08:50.719522 (+ 62us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:50.719591 (+ 69us) server_negotiation.cc:300] Negotiation successful
0903 08:08:50.719630 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":81,"threads_started":1}
I20250903 08:08:50.728997 21909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:50.726509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.193:44093 (local address 127.12.201.195:36249)
0903 08:08:50.726771 (+ 262us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:50.726775 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:50.726789 (+ 14us) server_negotiation.cc:408] Connection header received
0903 08:08:50.726918 (+ 129us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:50.726920 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:50.726968 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:50.727077 (+ 109us) server_negotiation.cc:227] Negotiated authn=TOKEN
0903 08:08:50.727399 (+ 322us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:50.727878 (+ 479us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:50.728479 (+ 601us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:50.728606 (+ 127us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:50.728643 (+ 37us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0903 08:08:50.728709 (+ 66us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0903 08:08:50.728764 (+ 55us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:50.728838 (+ 74us) server_negotiation.cc:300] Negotiation successful
0903 08:08:50.728881 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":96,"threads_started":1}
I20250903 08:08:50.728998 21908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:50.726414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.195:36249 (local address 127.12.201.193:44093)
0903 08:08:50.726733 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:50.726746 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:50.726828 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:50.727083 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:50.727085 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:50.727093 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0903 08:08:50.727292 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:50.727297 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:50.727965 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:50.727968 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:50.728388 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:50.728394 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:50.728484 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:50.728523 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0903 08:08:50.728773 (+ 250us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0903 08:08:50.728777 (+ 4us) client_negotiation.cc:769] Sending connection context
0903 08:08:50.728822 (+ 45us) client_negotiation.cc:240] Negotiation successful
0903 08:08:50.728882 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":122,"threads_started":1}
I20250903 08:08:50.734299 21602 raft_consensus.cc:1273] T a04dbc4a62c040f5bd7feae8d00b725d P 9bf6ef1f80704b49bf1b4354718df875 [term 1 FOLLOWER]: Refusing update from remote peer fa5050fb31d1413a949c35c167d78324: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 08:08:50.734401 21744 raft_consensus.cc:1273] T a04dbc4a62c040f5bd7feae8d00b725d P dc83179923a84bfaa56be4f55a7158ef [term 1 FOLLOWER]: Refusing update from remote peer fa5050fb31d1413a949c35c167d78324: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 08:08:50.734799 21902 consensus_queue.cc:1035] T a04dbc4a62c040f5bd7feae8d00b725d P fa5050fb31d1413a949c35c167d78324 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bf6ef1f80704b49bf1b4354718df875" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 46253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 08:08:50.734949 21825 consensus_queue.cc:1035] T a04dbc4a62c040f5bd7feae8d00b725d P fa5050fb31d1413a949c35c167d78324 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc83179923a84bfaa56be4f55a7158ef" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 36249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 08:08:50.751545 21914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:50.748739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.193:34563 (local address 127.12.201.195:47331)
0903 08:08:50.748984 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:50.748996 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:50.749096 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:50.749324 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:50.749327 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:50.749335 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0903 08:08:50.749556 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:50.749562 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:50.750261 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:50.750267 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:50.750710 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:50.750719 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:50.750836 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:50.750863 (+ 27us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0903 08:08:50.751273 (+ 410us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0903 08:08:50.751280 (+ 7us) client_negotiation.cc:769] Sending connection context
0903 08:08:50.751351 (+ 71us) client_negotiation.cc:240] Negotiation successful
0903 08:08:50.751432 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":79,"threads_started":1}
I20250903 08:08:50.751556 21905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:50.748797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.195:47331 (local address 127.12.201.193:34563)
0903 08:08:50.748923 (+ 126us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:50.748928 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:50.749065 (+ 137us) server_negotiation.cc:408] Connection header received
0903 08:08:50.749194 (+ 129us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:50.749197 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:50.749245 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:50.749308 (+ 63us) server_negotiation.cc:227] Negotiated authn=TOKEN
0903 08:08:50.749674 (+ 366us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:50.750154 (+ 480us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:50.750839 (+ 685us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:50.751065 (+ 226us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:50.751123 (+ 58us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0903 08:08:50.751200 (+ 77us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0903 08:08:50.751262 (+ 62us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:50.751372 (+ 110us) server_negotiation.cc:300] Negotiation successful
0903 08:08:50.751429 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20250903 08:08:50.771541 21301 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45702:
table { table_name: "test-table" } modify_external_catalogs: true
I20250903 08:08:50.771700 21301 catalog_manager.cc:2730] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45702:
table { table_name: "test-table" } modify_external_catalogs: true
I20250903 08:08:50.773689 21301 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3aa107cc15d641bca7f21dd4251fdf15: Sending DeleteTablet for 3 replicas of tablet a04dbc4a62c040f5bd7feae8d00b725d
I20250903 08:08:50.774267 21440 tablet_service.cc:1515] Processing DeleteTablet for tablet a04dbc4a62c040f5bd7feae8d00b725d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 08:08:50 UTC) from {username='oryx', principal='oryx/127.12.201.254@KRBTEST.COM'} at 127.0.0.1:54102
I20250903 08:08:50.774408 21724 tablet_service.cc:1515] Processing DeleteTablet for tablet a04dbc4a62c040f5bd7feae8d00b725d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 08:08:50 UTC) from {username='oryx', principal='oryx/127.12.201.254@KRBTEST.COM'} at 127.0.0.1:42216
I20250903 08:08:50.774408 21582 tablet_service.cc:1515] Processing DeleteTablet for tablet a04dbc4a62c040f5bd7feae8d00b725d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 08:08:50 UTC) from {username='oryx', principal='oryx/127.12.201.254@KRBTEST.COM'} at 127.0.0.1:49330
I20250903 08:08:50.774459 21842 tablet_replica.cc:331] T a04dbc4a62c040f5bd7feae8d00b725d P fa5050fb31d1413a949c35c167d78324: stopping tablet replica
I20250903 08:08:50.774590 21842 raft_consensus.cc:2241] T a04dbc4a62c040f5bd7feae8d00b725d P fa5050fb31d1413a949c35c167d78324 [term 1 LEADER]: Raft consensus shutting down.
I20250903 08:08:50.774618 21841 tablet_replica.cc:331] T a04dbc4a62c040f5bd7feae8d00b725d P 9bf6ef1f80704b49bf1b4354718df875: stopping tablet replica
I20250903 08:08:50.774735 21841 raft_consensus.cc:2241] T a04dbc4a62c040f5bd7feae8d00b725d P 9bf6ef1f80704b49bf1b4354718df875 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 08:08:50.774787 21842 raft_consensus.cc:2270] T a04dbc4a62c040f5bd7feae8d00b725d P fa5050fb31d1413a949c35c167d78324 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 08:08:50.774858 21841 raft_consensus.cc:2270] T a04dbc4a62c040f5bd7feae8d00b725d P 9bf6ef1f80704b49bf1b4354718df875 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 08:08:50.775055 21843 tablet_replica.cc:331] T a04dbc4a62c040f5bd7feae8d00b725d P dc83179923a84bfaa56be4f55a7158ef: stopping tablet replica
I20250903 08:08:50.775143 21843 raft_consensus.cc:2241] T a04dbc4a62c040f5bd7feae8d00b725d P dc83179923a84bfaa56be4f55a7158ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 08:08:50.775238 13095 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUML90r/build/debug/bin/kudu with pid 21375
I20250903 08:08:50.775259 21843 raft_consensus.cc:2270] T a04dbc4a62c040f5bd7feae8d00b725d P dc83179923a84bfaa56be4f55a7158ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 08:08:50.775573 21842 ts_tablet_manager.cc:1905] T a04dbc4a62c040f5bd7feae8d00b725d P fa5050fb31d1413a949c35c167d78324: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 08:08:50.776572 21843 ts_tablet_manager.cc:1905] T a04dbc4a62c040f5bd7feae8d00b725d P dc83179923a84bfaa56be4f55a7158ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 08:08:50.777156 21841 ts_tablet_manager.cc:1905] T a04dbc4a62c040f5bd7feae8d00b725d P 9bf6ef1f80704b49bf1b4354718df875: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 08:08:50.779127 21841 ts_tablet_manager.cc:1918] T a04dbc4a62c040f5bd7feae8d00b725d P 9bf6ef1f80704b49bf1b4354718df875: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20250903 08:08:50.779197 21841 log.cc:1199] T a04dbc4a62c040f5bd7feae8d00b725d P 9bf6ef1f80704b49bf1b4354718df875: Deleting WAL directory at /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/minicluster-data/ts-1/wal/wals/a04dbc4a62c040f5bd7feae8d00b725d
I20250903 08:08:50.779423 21841 ts_tablet_manager.cc:1939] T a04dbc4a62c040f5bd7feae8d00b725d P 9bf6ef1f80704b49bf1b4354718df875: Deleting consensus metadata
I20250903 08:08:50.779697 21843 ts_tablet_manager.cc:1918] T a04dbc4a62c040f5bd7feae8d00b725d P dc83179923a84bfaa56be4f55a7158ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20250903 08:08:50.779785 21843 log.cc:1199] T a04dbc4a62c040f5bd7feae8d00b725d P dc83179923a84bfaa56be4f55a7158ef: Deleting WAL directory at /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756886860253814-13095-0/minicluster-data/ts-2/wal/wals/a04dbc4a62c040f5bd7feae8d00b725d
I20250903 08:08:50.780067 21843 ts_tablet_manager.cc:1939] T a04dbc4a62c040f5bd7feae8d00b725d P dc83179923a84bfaa56be4f55a7158ef: Deleting consensus metadata
I20250903 08:08:50.780450 21286 catalog_manager.cc:4928] TS 9bf6ef1f80704b49bf1b4354718df875 (127.12.201.194:46253): tablet a04dbc4a62c040f5bd7feae8d00b725d (table test-table [id=3839c6f66f194c2b992323bf68b2ffce]) successfully deleted
I20250903 08:08:50.780834 21285 catalog_manager.cc:4928] TS dc83179923a84bfaa56be4f55a7158ef (127.12.201.195:36249): tablet a04dbc4a62c040f5bd7feae8d00b725d (table test-table [id=3839c6f66f194c2b992323bf68b2ffce]) successfully deleted
W20250903 08:08:50.785943 21288 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.201.193:34563 (error 108)
I20250903 08:08:50.786669 13095 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUML90r/build/debug/bin/kudu with pid 21517
I20250903 08:08:50.787122 21917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:50.786646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.193:34563 (local address 127.0.0.1:54118)
0903 08:08:50.786911 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:50.786999 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.201.193:34563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":54,"threads_started":1}
W20250903 08:08:50.787364 21288 catalog_manager.cc:4671] TS fa5050fb31d1413a949c35c167d78324 (127.12.201.193:34563): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a04dbc4a62c040f5bd7feae8d00b725d: Network error: Client connection negotiation failed: client connection to 127.12.201.193:34563: connect: Connection refused (error 111)
I20250903 08:08:50.793933 13095 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUML90r/build/debug/bin/kudu with pid 21659
I20250903 08:08:50.800896 13095 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUML90r/build/debug/bin/kudu with pid 21269
2025-09-03T08:08:50Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (4145 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1756886860253814-13095-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 03 08:08:50 dist-test-slave-27r9 krb5kdc[21920](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 03 08:08:50 dist-test-slave-27r9 krb5kdc[21920](info): set up 2 sockets
Sep 03 08:08:50 dist-test-slave-27r9 krb5kdc[21920](info): commencing operation
krb5kdc: starting...
W20250903 08:08:52.787796 13095 mini_kdc.cc:121] Time spent starting KDC: real 1.963s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1756886860253814-13095-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1756886860253814-13095-0/krb5kdc/test-user.keytab.
Sep 03 08:08:52 dist-test-slave-27r9 krb5kdc[21920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-09-03T08:08:52Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-03T08:08:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.201.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.201.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.201.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.201.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1756886860253814-13095-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.201.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.201.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.201.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.201.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1756886860253814-13095-0/krb5kdc/kudu.keytab.
I20250903 08:08:52.932273 13095 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUML90r/build/debug/bin/kudu
/tmp/dist-test-taskUML90r/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1756886860253814-13095-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1756886860253814-13095-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1756886860253814-13095-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1756886860253814-13095-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.201.254:39529
--webserver_interface=127.12.201.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.201.212:46425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.201.254:39529
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1756886860253814-13095-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1756886860253814-13095-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1756886860253814-13095-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 08:08:53.036441 21936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:08:53.036674 21936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:08:53.036731 21936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:08:53.039925 21936 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250903 08:08:53.039991 21936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 08:08:53.040014 21936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:08:53.040033 21936 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250903 08:08:53.040051 21936 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250903 08:08:53.044394 21936 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.201.212:46425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1756886860253814-13095-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1756886860253814-13095-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.201.254:39529
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.201.254:39529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1756886860253814-13095-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.201.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21936
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1756886860253814-13095-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 07:50:08 UTC on 5fd53c4cbb9d
build id 7874
I20250903 08:08:53.045629 21936 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:08:53.046437 21936 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:08:53.051766 21941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:53.051836 21942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:08:53.052031 21936 server_base.cc:1047] running on GCE node
W20250903 08:08:53.051844 21944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:08:53.052603 21936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:08:53.053476 21936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:08:53.054636 21936 hybrid_clock.cc:648] HybridClock initialized: now 1756886933054614 us; error 35 us; skew 500 ppm
W20250903 08:08:53.057778 21936 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2025-09-03T08:08:53Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (2256 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 03 08:08:53 dist-test-slave-27r9 krb5kdc[21949](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 03 08:08:53 dist-test-slave-27r9 krb5kdc[21949](info): set up 2 sockets
Sep 03 08:08:53 dist-test-slave-27r9 krb5kdc[21949](info): commencing operation
krb5kdc: starting...
W20250903 08:08:55.047825 13095 mini_kdc.cc:121] Time spent starting KDC: real 1.966s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/krb5kdc/test-user.keytab.
Sep 03 08:08:55 dist-test-slave-27r9 krb5kdc[21949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886935, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-09-03T08:08:55Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-03T08:08:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.201.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.201.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.201.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.201.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.201.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.201.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.201.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.201.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/krb5kdc/kudu.keytab.
I20250903 08:08:55.189483 13095 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUML90r/build/debug/bin/kudu
/tmp/dist-test-taskUML90r/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.201.254:42385
--webserver_interface=127.12.201.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.201.212:41737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.201.254:42385
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 08:08:55.296085 21965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:08:55.296321 21965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:08:55.296378 21965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:08:55.299588 21965 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250903 08:08:55.299659 21965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 08:08:55.299682 21965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:08:55.299702 21965 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250903 08:08:55.299719 21965 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250903 08:08:55.304162 21965 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.201.212:41737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.201.254:42385
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.201.254:42385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.201.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21965
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 07:50:08 UTC on 5fd53c4cbb9d
build id 7874
I20250903 08:08:55.305258 21965 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:08:55.306131 21965 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:08:55.311874 21973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:55.311865 21970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:55.311865 21971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:08:55.312426 21965 server_base.cc:1047] running on GCE node
I20250903 08:08:55.312943 21965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:08:55.313863 21965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:08:55.315065 21965 hybrid_clock.cc:648] HybridClock initialized: now 1756886935315045 us; error 35 us; skew 500 ppm
Sep 03 08:08:55 dist-test-slave-27r9 krb5kdc[21949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886935, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 08:08:55.317914 21965 init.cc:377] Logged in from keytab as kudu/127.12.201.254@KRBTEST.COM (short username kudu)
I20250903 08:08:55.319080 21965 webserver.cc:480] Webserver started at http://127.12.201.254:44721/ using document root <none> and password file <none>
I20250903 08:08:55.319625 21965 fs_manager.cc:362] Metadata directory not provided
I20250903 08:08:55.319674 21965 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:08:55.319836 21965 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:08:55.321571 21965 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/master-0/data/instance:
uuid: "3081f8b81aaf429985b108e3a9f92f6e"
format_stamp: "Formatted at 2025-09-03 08:08:55 on dist-test-slave-27r9"
server_key: "a2a9090c591c4239cc24f4f15230d7a6"
server_key_iv: "a2d50e4aac5f9beb0110f35ce0afe893"
server_key_version: "encryptionkey@0"
I20250903 08:08:55.322021 21965 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/master-0/wal/instance:
uuid: "3081f8b81aaf429985b108e3a9f92f6e"
format_stamp: "Formatted at 2025-09-03 08:08:55 on dist-test-slave-27r9"
server_key: "a2a9090c591c4239cc24f4f15230d7a6"
server_key_iv: "a2d50e4aac5f9beb0110f35ce0afe893"
server_key_version: "encryptionkey@0"
I20250903 08:08:55.325577 21965 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20250903 08:08:55.327880 21980 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:55.329003 21965 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250903 08:08:55.329110 21965 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/master-0/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/master-0/wal
uuid: "3081f8b81aaf429985b108e3a9f92f6e"
format_stamp: "Formatted at 2025-09-03 08:08:55 on dist-test-slave-27r9"
server_key: "a2a9090c591c4239cc24f4f15230d7a6"
server_key_iv: "a2d50e4aac5f9beb0110f35ce0afe893"
server_key_version: "encryptionkey@0"
I20250903 08:08:55.329195 21965 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:08:55.374439 21965 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:08:55.377192 21965 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:08:55.377341 21965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:08:55.384644 21965 rpc_server.cc:307] RPC server started. Bound to: 127.12.201.254:42385
I20250903 08:08:55.384635 22032 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.201.254:42385 every 8 connection(s)
I20250903 08:08:55.385705 21965 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/master-0/data/info.pb
I20250903 08:08:55.388236 22033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:08:55.393782 22033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3081f8b81aaf429985b108e3a9f92f6e: Bootstrap starting.
I20250903 08:08:55.395869 13095 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUML90r/build/debug/bin/kudu as pid 21965
I20250903 08:08:55.395975 13095 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/master-0/wal/instance
I20250903 08:08:55.396152 22033 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3081f8b81aaf429985b108e3a9f92f6e: Neither blocks nor log segments found. Creating new log.
I20250903 08:08:55.396229 13095 external_mini_cluster.cc:1468] Setting key 8883232673366813e60ededb781afd8c
I20250903 08:08:55.396974 22033 log.cc:826] T 00000000000000000000000000000000 P 3081f8b81aaf429985b108e3a9f92f6e: Log is configured to *not* fsync() on all Append() calls
I20250903 08:08:55.398808 22033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3081f8b81aaf429985b108e3a9f92f6e: No bootstrap required, opened a new log
I20250903 08:08:55.401243 22033 raft_consensus.cc:357] T 00000000000000000000000000000000 P 3081f8b81aaf429985b108e3a9f92f6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3081f8b81aaf429985b108e3a9f92f6e" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 42385 } }
I20250903 08:08:55.401479 22033 raft_consensus.cc:383] T 00000000000000000000000000000000 P 3081f8b81aaf429985b108e3a9f92f6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:08:55.401579 22033 raft_consensus.cc:738] T 00000000000000000000000000000000 P 3081f8b81aaf429985b108e3a9f92f6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3081f8b81aaf429985b108e3a9f92f6e, State: Initialized, Role: FOLLOWER
I20250903 08:08:55.402047 22033 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3081f8b81aaf429985b108e3a9f92f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3081f8b81aaf429985b108e3a9f92f6e" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 42385 } }
I20250903 08:08:55.402173 22033 raft_consensus.cc:397] T 00000000000000000000000000000000 P 3081f8b81aaf429985b108e3a9f92f6e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250903 08:08:55.402233 22033 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3081f8b81aaf429985b108e3a9f92f6e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250903 08:08:55.402340 22033 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 3081f8b81aaf429985b108e3a9f92f6e [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:08:55.403343 22033 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3081f8b81aaf429985b108e3a9f92f6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3081f8b81aaf429985b108e3a9f92f6e" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 42385 } }
Sep 03 08:08:55 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886935, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.201.254@KRBTEST.COM
I20250903 08:08:55.403717 22033 leader_election.cc:304] T 00000000000000000000000000000000 P 3081f8b81aaf429985b108e3a9f92f6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3081f8b81aaf429985b108e3a9f92f6e; no voters:
I20250903 08:08:55.404007 22033 leader_election.cc:290] T 00000000000000000000000000000000 P 3081f8b81aaf429985b108e3a9f92f6e [CANDIDATE]: Term 1 election: Requested vote from peers
I20250903 08:08:55.404152 22038 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 3081f8b81aaf429985b108e3a9f92f6e [term 1 FOLLOWER]: Leader election won for term 1
I20250903 08:08:55.404356 22038 raft_consensus.cc:695] T 00000000000000000000000000000000 P 3081f8b81aaf429985b108e3a9f92f6e [term 1 LEADER]: Becoming Leader. State: Replica: 3081f8b81aaf429985b108e3a9f92f6e, State: Running, Role: LEADER
I20250903 08:08:55.404688 22038 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3081f8b81aaf429985b108e3a9f92f6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3081f8b81aaf429985b108e3a9f92f6e" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 42385 } }
I20250903 08:08:55.405169 22033 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3081f8b81aaf429985b108e3a9f92f6e [sys.catalog]: configured and running, proceeding with master startup.
I20250903 08:08:55.405997 22038 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3081f8b81aaf429985b108e3a9f92f6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3081f8b81aaf429985b108e3a9f92f6e. Latest consensus state: current_term: 1 leader_uuid: "3081f8b81aaf429985b108e3a9f92f6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3081f8b81aaf429985b108e3a9f92f6e" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 42385 } } }
I20250903 08:08:55.406224 22038 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3081f8b81aaf429985b108e3a9f92f6e [sys.catalog]: This master's current role is: LEADER
I20250903 08:08:55.406651 22047 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250903 08:08:55.408078 22039 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3081f8b81aaf429985b108e3a9f92f6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3081f8b81aaf429985b108e3a9f92f6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3081f8b81aaf429985b108e3a9f92f6e" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 42385 } } }
I20250903 08:08:55.408246 22039 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3081f8b81aaf429985b108e3a9f92f6e [sys.catalog]: This master's current role is: LEADER
I20250903 08:08:55.410005 22047 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250903 08:08:55.410382 22037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:55.397319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54732 (local address 127.12.201.254:42385)
0903 08:08:55.397793 (+ 474us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:55.397804 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:55.397841 (+ 37us) server_negotiation.cc:408] Connection header received
0903 08:08:55.398372 (+ 531us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:55.398389 (+ 17us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:55.398684 (+ 295us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:55.399030 (+ 346us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:55.399915 (+ 885us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:55.400653 (+ 738us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:55.401327 (+ 674us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:55.401623 (+ 296us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:55.404175 (+ 2552us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:55.404200 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:55.404218 (+ 18us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:08:55.404264 (+ 46us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:55.407186 (+ 2922us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:55.407868 (+ 682us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:55.407874 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:55.407879 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:55.407988 (+ 109us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:55.408302 (+ 314us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:55.408307 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:55.408310 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:55.408710 (+ 400us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:55.408911 (+ 201us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:55.409380 (+ 469us) server_negotiation.cc:300] Negotiation successful
0903 08:08:55.409645 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":165,"threads_started":1}
I20250903 08:08:55.416042 22047 catalog_manager.cc:1349] Generated new cluster ID: 8735117e5924418bbe93e94191545355
I20250903 08:08:55.416145 22047 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250903 08:08:55.422606 22047 catalog_manager.cc:1372] Generated new certificate authority record
I20250903 08:08:55.423578 22047 catalog_manager.cc:1506] Loading token signing keys...
I20250903 08:08:55.432294 22047 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 3081f8b81aaf429985b108e3a9f92f6e: Generated new TSK 0
I20250903 08:08:55.433053 22047 catalog_manager.cc:1516] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.201.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.201.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.201.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.201.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.201.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.201.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.201.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.201.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/krb5kdc/kudu.keytab.
I20250903 08:08:55.485914 13095 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUML90r/build/debug/bin/kudu
/tmp/dist-test-taskUML90r/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.201.193:0
--local_ip_for_outbound_sockets=127.12.201.193
--webserver_interface=127.12.201.193
--webserver_port=0
--tserver_master_addrs=127.12.201.254:42385
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.201.212:41737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 08:08:55.591986 22061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:08:55.592245 22061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:08:55.592339 22061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:08:55.595630 22061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 08:08:55.595705 22061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:08:55.595826 22061 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.201.193
I20250903 08:08:55.600183 22061 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.201.212:41737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.201.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.201.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.12.201.254:42385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22061
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.201.193
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 07:50:08 UTC on 5fd53c4cbb9d
build id 7874
I20250903 08:08:55.601285 22061 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:08:55.602206 22061 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:08:55.608732 22069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:55.608902 22066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:55.609005 22067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:08:55.609161 22061 server_base.cc:1047] running on GCE node
I20250903 08:08:55.609535 22061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:08:55.610085 22061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:08:55.611243 22061 hybrid_clock.cc:648] HybridClock initialized: now 1756886935611219 us; error 39 us; skew 500 ppm
Sep 03 08:08:55 dist-test-slave-27r9 krb5kdc[21949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886935, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 08:08:55.614076 22061 init.cc:377] Logged in from keytab as kudu/127.12.201.193@KRBTEST.COM (short username kudu)
I20250903 08:08:55.615160 22061 webserver.cc:480] Webserver started at http://127.12.201.193:38515/ using document root <none> and password file <none>
I20250903 08:08:55.615686 22061 fs_manager.cc:362] Metadata directory not provided
I20250903 08:08:55.615733 22061 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:08:55.615891 22061 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:08:55.617660 22061 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-0/data/instance:
uuid: "efa9f2d3f2274aefbe3b8b25325ef7eb"
format_stamp: "Formatted at 2025-09-03 08:08:55 on dist-test-slave-27r9"
server_key: "136f66818a490118f9bf614ebd08eaaf"
server_key_iv: "9205f73ad3ac1998a8f505b981ee8c2b"
server_key_version: "encryptionkey@0"
I20250903 08:08:55.618105 22061 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-0/wal/instance:
uuid: "efa9f2d3f2274aefbe3b8b25325ef7eb"
format_stamp: "Formatted at 2025-09-03 08:08:55 on dist-test-slave-27r9"
server_key: "136f66818a490118f9bf614ebd08eaaf"
server_key_iv: "9205f73ad3ac1998a8f505b981ee8c2b"
server_key_version: "encryptionkey@0"
I20250903 08:08:55.621528 22061 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20250903 08:08:55.623801 22076 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:55.625023 22061 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250903 08:08:55.625125 22061 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-0/wal
uuid: "efa9f2d3f2274aefbe3b8b25325ef7eb"
format_stamp: "Formatted at 2025-09-03 08:08:55 on dist-test-slave-27r9"
server_key: "136f66818a490118f9bf614ebd08eaaf"
server_key_iv: "9205f73ad3ac1998a8f505b981ee8c2b"
server_key_version: "encryptionkey@0"
I20250903 08:08:55.625213 22061 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:08:55.640055 22061 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:08:55.643213 22061 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:08:55.643383 22061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:08:55.643914 22061 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 08:08:55.644719 22061 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 08:08:55.644766 22061 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:55.644817 22061 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 08:08:55.644833 22061 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:55.653391 22061 rpc_server.cc:307] RPC server started. Bound to: 127.12.201.193:46833
I20250903 08:08:55.653426 22189 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.201.193:46833 every 8 connection(s)
I20250903 08:08:55.654340 22061 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-0/data/info.pb
I20250903 08:08:55.662055 13095 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUML90r/build/debug/bin/kudu as pid 22061
I20250903 08:08:55.662191 13095 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-0/wal/instance
I20250903 08:08:55.662446 13095 external_mini_cluster.cc:1468] Setting key 39454caba0632b32d3954b649722c085
Sep 03 08:08:55 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886935, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.193@KRBTEST.COM for kudu/127.12.201.254@KRBTEST.COM
I20250903 08:08:55.668172 22037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:55.656189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.193:50775 (local address 127.12.201.254:42385)
0903 08:08:55.656335 (+ 146us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:55.656340 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:55.657077 (+ 737us) server_negotiation.cc:408] Connection header received
0903 08:08:55.658054 (+ 977us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:55.658057 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:55.658107 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:55.658182 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:55.659465 (+ 1283us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:55.659982 (+ 517us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:55.661153 (+ 1171us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:55.661312 (+ 159us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:55.663681 (+ 2369us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:55.663698 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:55.663701 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:08:55.663738 (+ 37us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:55.665690 (+ 1952us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:55.666452 (+ 762us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:55.666455 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:55.666457 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:55.666512 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:55.666878 (+ 366us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:55.666884 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:55.666889 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:55.667144 (+ 255us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:55.667254 (+ 110us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:55.667760 (+ 506us) server_negotiation.cc:300] Negotiation successful
0903 08:08:55.667935 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20250903 08:08:55.668717 22192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:55.656463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:42385 (local address 127.12.201.193:50775)
0903 08:08:55.656913 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:55.656954 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:55.657849 (+ 895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:55.658328 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:55.658336 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:55.658734 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:08:55.659328 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:55.659340 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:55.660120 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:55.660123 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:55.661019 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:55.661028 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:55.661211 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:55.661801 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:08:55.661822 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:08:55.663499 (+ 1677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:08:55.665844 (+ 2345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:55.665852 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:55.665875 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:55.666319 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:55.666609 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:55.666615 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:55.666620 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:55.666738 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:55.667251 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:08:55.667256 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:08:55.667498 (+ 242us) client_negotiation.cc:769] Sending connection context
0903 08:08:55.667680 (+ 182us) client_negotiation.cc:240] Negotiation successful
0903 08:08:55.667930 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":98,"threads_started":1}
I20250903 08:08:55.669987 22190 heartbeater.cc:344] Connected to a master server at 127.12.201.254:42385
I20250903 08:08:55.670236 22190 heartbeater.cc:461] Registering TS with master...
I20250903 08:08:55.670769 22190 heartbeater.cc:507] Master 127.12.201.254:42385 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.12.201.194@KRBTEST.COM; defaulting to no policy
I20250903 08:08:55.672415 21997 ts_manager.cc:194] Registered new tserver with Master: efa9f2d3f2274aefbe3b8b25325ef7eb (127.12.201.193:46833)
I20250903 08:08:55.673568 21997 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.201.193@KRBTEST.COM'} at 127.12.201.193:50775
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.201.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.201.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.201.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.201.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.201.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.201.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.201.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/krb5kdc/kudu.keytab.
I20250903 08:08:55.713579 13095 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUML90r/build/debug/bin/kudu
/tmp/dist-test-taskUML90r/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.201.194:0
--local_ip_for_outbound_sockets=127.12.201.194
--webserver_interface=127.12.201.194
--webserver_port=0
--tserver_master_addrs=127.12.201.254:42385
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.201.212:41737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 08:08:55.820122 22197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:08:55.820355 22197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:08:55.820415 22197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:08:55.823704 22197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 08:08:55.823772 22197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:08:55.823855 22197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.201.194
I20250903 08:08:55.828136 22197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.201.212:41737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.201.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.201.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.12.201.254:42385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.201.194
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 07:50:08 UTC on 5fd53c4cbb9d
build id 7874
I20250903 08:08:55.829242 22197 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:08:55.830102 22197 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:08:55.836756 22205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:55.836755 22202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:55.836755 22203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:08:55.837042 22197 server_base.cc:1047] running on GCE node
I20250903 08:08:55.837424 22197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:08:55.837985 22197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:08:55.839138 22197 hybrid_clock.cc:648] HybridClock initialized: now 1756886935839117 us; error 41 us; skew 500 ppm
Sep 03 08:08:55 dist-test-slave-27r9 krb5kdc[21949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886935, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 08:08:55.841951 22197 init.cc:377] Logged in from keytab as kudu/127.12.201.194@KRBTEST.COM (short username kudu)
I20250903 08:08:55.843119 22197 webserver.cc:480] Webserver started at http://127.12.201.194:38001/ using document root <none> and password file <none>
I20250903 08:08:55.843698 22197 fs_manager.cc:362] Metadata directory not provided
I20250903 08:08:55.843750 22197 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:08:55.843936 22197 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:08:55.845696 22197 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-1/data/instance:
uuid: "4c5720a5742d4c58800e541283fbaffd"
format_stamp: "Formatted at 2025-09-03 08:08:55 on dist-test-slave-27r9"
server_key: "518b6f20e91954f07c2439cb4850127d"
server_key_iv: "1511f76ed720e2ecca9d0f317c481d17"
server_key_version: "encryptionkey@0"
I20250903 08:08:55.846177 22197 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-1/wal/instance:
uuid: "4c5720a5742d4c58800e541283fbaffd"
format_stamp: "Formatted at 2025-09-03 08:08:55 on dist-test-slave-27r9"
server_key: "518b6f20e91954f07c2439cb4850127d"
server_key_iv: "1511f76ed720e2ecca9d0f317c481d17"
server_key_version: "encryptionkey@0"
I20250903 08:08:55.849639 22197 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20250903 08:08:55.852001 22212 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:55.853111 22197 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250903 08:08:55.853231 22197 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-1/wal
uuid: "4c5720a5742d4c58800e541283fbaffd"
format_stamp: "Formatted at 2025-09-03 08:08:55 on dist-test-slave-27r9"
server_key: "518b6f20e91954f07c2439cb4850127d"
server_key_iv: "1511f76ed720e2ecca9d0f317c481d17"
server_key_version: "encryptionkey@0"
I20250903 08:08:55.853331 22197 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:08:55.875629 22197 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:08:55.878674 22197 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:08:55.878891 22197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:08:55.879498 22197 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 08:08:55.880375 22197 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 08:08:55.880429 22197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:55.880491 22197 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 08:08:55.880517 22197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:55.889429 22197 rpc_server.cc:307] RPC server started. Bound to: 127.12.201.194:41023
I20250903 08:08:55.889469 22325 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.201.194:41023 every 8 connection(s)
I20250903 08:08:55.890381 22197 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-1/data/info.pb
Sep 03 08:08:55 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886935, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.194@KRBTEST.COM for kudu/127.12.201.254@KRBTEST.COM
I20250903 08:08:55.899577 13095 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUML90r/build/debug/bin/kudu as pid 22197
I20250903 08:08:55.899695 13095 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-1/wal/instance
I20250903 08:08:55.899935 13095 external_mini_cluster.cc:1468] Setting key 7ba1450ac3337eda560e13e1627a3857
I20250903 08:08:55.902602 22037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:55.892153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.194:49061 (local address 127.12.201.254:42385)
0903 08:08:55.892294 (+ 141us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:55.892297 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:55.892921 (+ 624us) server_negotiation.cc:408] Connection header received
0903 08:08:55.893723 (+ 802us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:55.893726 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:55.893774 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:55.893848 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:55.895075 (+ 1227us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:55.895567 (+ 492us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:55.896632 (+ 1065us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:55.896781 (+ 149us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:55.899102 (+ 2321us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:55.899122 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:55.899126 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:08:55.899153 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:55.900785 (+ 1632us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:55.901309 (+ 524us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:55.901313 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:55.901315 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:55.901360 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:55.901606 (+ 246us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:55.901610 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:55.901612 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:55.901755 (+ 143us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:55.901843 (+ 88us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:55.902314 (+ 471us) server_negotiation.cc:300] Negotiation successful
0903 08:08:55.902436 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20250903 08:08:55.903335 22328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:55.892387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:42385 (local address 127.12.201.194:49061)
0903 08:08:55.892780 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:55.892816 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:55.893548 (+ 732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:55.893987 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:55.893994 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:55.894379 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:08:55.894915 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:55.894928 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:55.895657 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:55.895660 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:55.896506 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:55.896513 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:55.896663 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:55.897247 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:08:55.897268 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:08:55.898924 (+ 1656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:08:55.900909 (+ 1985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:55.900914 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:55.900923 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:55.901222 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:55.901438 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:55.901441 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:55.901443 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:55.901525 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:55.901848 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:08:55.901858 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:08:55.902086 (+ 228us) client_negotiation.cc:769] Sending connection context
0903 08:08:55.902269 (+ 183us) client_negotiation.cc:240] Negotiation successful
0903 08:08:55.902527 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":98,"threads_started":1}
I20250903 08:08:55.904316 22326 heartbeater.cc:344] Connected to a master server at 127.12.201.254:42385
I20250903 08:08:55.904548 22326 heartbeater.cc:461] Registering TS with master...
I20250903 08:08:55.905043 22326 heartbeater.cc:507] Master 127.12.201.254:42385 requested a full tablet report, sending...
I20250903 08:08:55.906024 21997 ts_manager.cc:194] Registered new tserver with Master: 4c5720a5742d4c58800e541283fbaffd (127.12.201.194:41023)
I20250903 08:08:55.906797 21997 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.201.194@KRBTEST.COM'} at 127.12.201.194:49061
WARNING: no policy specified for kudu/127.12.201.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.201.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.201.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.201.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.201.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.201.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.201.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.201.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/krb5kdc/kudu.keytab.
I20250903 08:08:55.947842 13095 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUML90r/build/debug/bin/kudu
/tmp/dist-test-taskUML90r/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.201.195:0
--local_ip_for_outbound_sockets=127.12.201.195
--webserver_interface=127.12.201.195
--webserver_port=0
--tserver_master_addrs=127.12.201.254:42385
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.201.212:41737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 08:08:56.052145 22333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:08:56.052340 22333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:08:56.052415 22333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:08:56.055562 22333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 08:08:56.055617 22333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:08:56.055693 22333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.201.195
I20250903 08:08:56.061344 22333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.201.212:41737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.201.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.201.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.12.201.254:42385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.201.195
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 07:50:08 UTC on 5fd53c4cbb9d
build id 7874
I20250903 08:08:56.062683 22333 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:08:56.063897 22333 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:08:56.070778 22341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:08:56.070832 22333 server_base.cc:1047] running on GCE node
W20250903 08:08:56.070775 22339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:56.070770 22338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:08:56.071367 22333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:08:56.071910 22333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:08:56.073053 22333 hybrid_clock.cc:648] HybridClock initialized: now 1756886936073030 us; error 39 us; skew 500 ppm
Sep 03 08:08:56 dist-test-slave-27r9 krb5kdc[21949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886936, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 08:08:56.075832 22333 init.cc:377] Logged in from keytab as kudu/127.12.201.195@KRBTEST.COM (short username kudu)
I20250903 08:08:56.076858 22333 webserver.cc:480] Webserver started at http://127.12.201.195:38421/ using document root <none> and password file <none>
I20250903 08:08:56.077412 22333 fs_manager.cc:362] Metadata directory not provided
I20250903 08:08:56.077466 22333 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:08:56.077649 22333 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:08:56.079406 22333 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-2/data/instance:
uuid: "fc2c501335b34acb8f4ef6df0accf5c8"
format_stamp: "Formatted at 2025-09-03 08:08:56 on dist-test-slave-27r9"
server_key: "962d79e7297d9f14b9bf5d7e802f1560"
server_key_iv: "a18547766b4812a9cf6bb54692576a80"
server_key_version: "encryptionkey@0"
I20250903 08:08:56.079864 22333 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-2/wal/instance:
uuid: "fc2c501335b34acb8f4ef6df0accf5c8"
format_stamp: "Formatted at 2025-09-03 08:08:56 on dist-test-slave-27r9"
server_key: "962d79e7297d9f14b9bf5d7e802f1560"
server_key_iv: "a18547766b4812a9cf6bb54692576a80"
server_key_version: "encryptionkey@0"
I20250903 08:08:56.083098 22333 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250903 08:08:56.085096 22348 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:56.086050 22333 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250903 08:08:56.086159 22333 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-2/wal
uuid: "fc2c501335b34acb8f4ef6df0accf5c8"
format_stamp: "Formatted at 2025-09-03 08:08:56 on dist-test-slave-27r9"
server_key: "962d79e7297d9f14b9bf5d7e802f1560"
server_key_iv: "a18547766b4812a9cf6bb54692576a80"
server_key_version: "encryptionkey@0"
I20250903 08:08:56.086241 22333 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:08:56.098732 22333 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:08:56.101544 22333 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:08:56.101719 22333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:08:56.102232 22333 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 08:08:56.103102 22333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 08:08:56.103161 22333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:56.103228 22333 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 08:08:56.103258 22333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:56.111744 22333 rpc_server.cc:307] RPC server started. Bound to: 127.12.201.195:41079
I20250903 08:08:56.111766 22461 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.201.195:41079 every 8 connection(s)
I20250903 08:08:56.112677 22333 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-2/data/info.pb
I20250903 08:08:56.113896 13095 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUML90r/build/debug/bin/kudu as pid 22333
I20250903 08:08:56.114007 13095 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-2/wal/instance
I20250903 08:08:56.114259 13095 external_mini_cluster.cc:1468] Setting key bc0753cd0357b53e93957754aa053f4a
Sep 03 08:08:56 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886936, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.195@KRBTEST.COM for kudu/127.12.201.254@KRBTEST.COM
I20250903 08:08:56.124369 22037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:56.114422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.195:43249 (local address 127.12.201.254:42385)
0903 08:08:56.114582 (+ 160us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:56.114587 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:56.115345 (+ 758us) server_negotiation.cc:408] Connection header received
0903 08:08:56.116262 (+ 917us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:56.116266 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:56.116319 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:56.116386 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:56.117551 (+ 1165us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:56.118026 (+ 475us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:56.119124 (+ 1098us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:56.119256 (+ 132us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:56.121277 (+ 2021us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:56.121292 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:56.121295 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:08:56.121320 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:56.122721 (+ 1401us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:56.123199 (+ 478us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:56.123202 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:56.123203 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:56.123242 (+ 39us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:56.123509 (+ 267us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:56.123511 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:56.123512 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:56.123646 (+ 134us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:56.123713 (+ 67us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:56.124160 (+ 447us) server_negotiation.cc:300] Negotiation successful
0903 08:08:56.124250 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20250903 08:08:56.124986 22464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:56.114697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:42385 (local address 127.12.201.195:43249)
0903 08:08:56.115202 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:56.115252 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:56.116043 (+ 791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:56.116524 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:56.116532 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:56.116910 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:08:56.117438 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:56.117450 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:56.118125 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:56.118127 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:56.119034 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:56.119043 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:56.119198 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:56.119744 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:08:56.119768 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:08:56.121144 (+ 1376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:08:56.122825 (+ 1681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:56.122829 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:56.122838 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:56.123120 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:56.123320 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:08:56.123326 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:08:56.123330 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:08:56.123435 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:08:56.123722 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:08:56.123727 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:08:56.123954 (+ 227us) client_negotiation.cc:769] Sending connection context
0903 08:08:56.124113 (+ 159us) client_negotiation.cc:240] Negotiation successful
0903 08:08:56.124322 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":138,"threads_started":1}
I20250903 08:08:56.125993 22462 heartbeater.cc:344] Connected to a master server at 127.12.201.254:42385
I20250903 08:08:56.126222 22462 heartbeater.cc:461] Registering TS with master...
I20250903 08:08:56.126713 22462 heartbeater.cc:507] Master 127.12.201.254:42385 requested a full tablet report, sending...
I20250903 08:08:56.127719 21996 ts_manager.cc:194] Registered new tserver with Master: fc2c501335b34acb8f4ef6df0accf5c8 (127.12.201.195:41079)
I20250903 08:08:56.128296 21996 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.201.195@KRBTEST.COM'} at 127.12.201.195:43249
I20250903 08:08:56.139719 13095 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.12.201.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.201.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.201.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/krb5kdc/oryx_127.12.201.193.keytab.
Entry for principal oryx/127.12.201.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/krb5kdc/oryx_127.12.201.193.keytab.
WARNING: no policy specified for oryx/127.12.201.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.201.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.201.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/krb5kdc/oryx_127.12.201.194.keytab.
Entry for principal oryx/127.12.201.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/krb5kdc/oryx_127.12.201.194.keytab.
WARNING: no policy specified for oryx/127.12.201.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.201.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.201.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/krb5kdc/oryx_127.12.201.195.keytab.
Entry for principal oryx/127.12.201.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/krb5kdc/oryx_127.12.201.195.keytab.
I20250903 08:08:56.209203 13095 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUML90r/build/debug/bin/kudu with pid 22061
I20250903 08:08:56.214866 13095 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUML90r/build/debug/bin/kudu with pid 22197
I20250903 08:08:56.220329 13095 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUML90r/build/debug/bin/kudu with pid 22333
I20250903 08:08:56.225685 13095 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUML90r/build/debug/bin/kudu with pid 21965
I20250903 08:08:56.232002 13095 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUML90r/build/debug/bin/kudu
/tmp/dist-test-taskUML90r/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.201.254:42385
--webserver_interface=127.12.201.254
--webserver_port=44721
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.201.212:41737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.201.254:42385
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 08:08:56.336944 22471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:08:56.337182 22471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:08:56.337247 22471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:08:56.340513 22471 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250903 08:08:56.340582 22471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 08:08:56.340605 22471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:08:56.340626 22471 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250903 08:08:56.340664 22471 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250903 08:08:56.344962 22471 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.201.212:41737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.201.254:42385
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.201.254:42385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.201.254
--webserver_port=44721
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22471
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 07:50:08 UTC on 5fd53c4cbb9d
build id 7874
I20250903 08:08:56.346094 22471 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:08:56.346961 22471 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:08:56.352501 22477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:56.352506 22479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:08:56.352679 22471 server_base.cc:1047] running on GCE node
W20250903 08:08:56.352504 22476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:08:56.353278 22471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:08:56.354143 22471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:08:56.355322 22471 hybrid_clock.cc:648] HybridClock initialized: now 1756886936355305 us; error 31 us; skew 500 ppm
Sep 03 08:08:56 dist-test-slave-27r9 krb5kdc[21949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886936, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 08:08:56.358296 22471 init.cc:377] Logged in from keytab as kudu/127.12.201.254@KRBTEST.COM (short username kudu)
I20250903 08:08:56.359470 22471 webserver.cc:480] Webserver started at http://127.12.201.254:44721/ using document root <none> and password file <none>
I20250903 08:08:56.360050 22471 fs_manager.cc:362] Metadata directory not provided
I20250903 08:08:56.360116 22471 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:08:56.363651 22471 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20250903 08:08:56.365557 22486 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:56.366521 22471 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250903 08:08:56.366632 22471 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/master-0/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/master-0/wal
uuid: "3081f8b81aaf429985b108e3a9f92f6e"
format_stamp: "Formatted at 2025-09-03 08:08:55 on dist-test-slave-27r9"
server_key: "a2a9090c591c4239cc24f4f15230d7a6"
server_key_iv: "a2d50e4aac5f9beb0110f35ce0afe893"
server_key_version: "encryptionkey@0"
I20250903 08:08:56.367074 22471 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:08:56.393214 22471 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:08:56.396289 22471 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:08:56.396478 22471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:08:56.404165 22538 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.201.254:42385 every 8 connection(s)
I20250903 08:08:56.404177 22471 rpc_server.cc:307] RPC server started. Bound to: 127.12.201.254:42385
I20250903 08:08:56.405362 22471 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/master-0/data/info.pb
I20250903 08:08:56.407598 13095 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUML90r/build/debug/bin/kudu as pid 22471
I20250903 08:08:56.408061 13095 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUML90r/build/debug/bin/kudu
/tmp/dist-test-taskUML90r/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.201.193:46833
--local_ip_for_outbound_sockets=127.12.201.193
--tserver_master_addrs=127.12.201.254:42385
--webserver_port=38515
--webserver_interface=127.12.201.193
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.201.212:41737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.12.201.193
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/krb5kdc/oryx_127.12.201.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20250903 08:08:56.409451 22539 sys_catalog.cc:263] Verifying existing consensus state
I20250903 08:08:56.411072 22539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3081f8b81aaf429985b108e3a9f92f6e: Bootstrap starting.
I20250903 08:08:56.427253 22539 log.cc:826] T 00000000000000000000000000000000 P 3081f8b81aaf429985b108e3a9f92f6e: Log is configured to *not* fsync() on all Append() calls
I20250903 08:08:56.433262 22539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3081f8b81aaf429985b108e3a9f92f6e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250903 08:08:56.433621 22539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3081f8b81aaf429985b108e3a9f92f6e: Bootstrap complete.
I20250903 08:08:56.436987 22539 raft_consensus.cc:357] T 00000000000000000000000000000000 P 3081f8b81aaf429985b108e3a9f92f6e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3081f8b81aaf429985b108e3a9f92f6e" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 42385 } }
I20250903 08:08:56.437286 22539 raft_consensus.cc:738] T 00000000000000000000000000000000 P 3081f8b81aaf429985b108e3a9f92f6e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3081f8b81aaf429985b108e3a9f92f6e, State: Initialized, Role: FOLLOWER
I20250903 08:08:56.437734 22539 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3081f8b81aaf429985b108e3a9f92f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3081f8b81aaf429985b108e3a9f92f6e" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 42385 } }
I20250903 08:08:56.437844 22539 raft_consensus.cc:397] T 00000000000000000000000000000000 P 3081f8b81aaf429985b108e3a9f92f6e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250903 08:08:56.437919 22539 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3081f8b81aaf429985b108e3a9f92f6e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250903 08:08:56.438045 22539 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 3081f8b81aaf429985b108e3a9f92f6e [term 1 FOLLOWER]: Advancing to term 2
I20250903 08:08:56.439649 22539 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3081f8b81aaf429985b108e3a9f92f6e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3081f8b81aaf429985b108e3a9f92f6e" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 42385 } }
I20250903 08:08:56.439977 22539 leader_election.cc:304] T 00000000000000000000000000000000 P 3081f8b81aaf429985b108e3a9f92f6e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3081f8b81aaf429985b108e3a9f92f6e; no voters:
I20250903 08:08:56.440296 22539 leader_election.cc:290] T 00000000000000000000000000000000 P 3081f8b81aaf429985b108e3a9f92f6e [CANDIDATE]: Term 2 election: Requested vote from peers
I20250903 08:08:56.440392 22543 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 3081f8b81aaf429985b108e3a9f92f6e [term 2 FOLLOWER]: Leader election won for term 2
I20250903 08:08:56.440621 22543 raft_consensus.cc:695] T 00000000000000000000000000000000 P 3081f8b81aaf429985b108e3a9f92f6e [term 2 LEADER]: Becoming Leader. State: Replica: 3081f8b81aaf429985b108e3a9f92f6e, State: Running, Role: LEADER
I20250903 08:08:56.440985 22543 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3081f8b81aaf429985b108e3a9f92f6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3081f8b81aaf429985b108e3a9f92f6e" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 42385 } }
I20250903 08:08:56.441457 22539 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3081f8b81aaf429985b108e3a9f92f6e [sys.catalog]: configured and running, proceeding with master startup.
I20250903 08:08:56.442354 22545 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3081f8b81aaf429985b108e3a9f92f6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3081f8b81aaf429985b108e3a9f92f6e. Latest consensus state: current_term: 2 leader_uuid: "3081f8b81aaf429985b108e3a9f92f6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3081f8b81aaf429985b108e3a9f92f6e" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 42385 } } }
I20250903 08:08:56.442497 22545 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3081f8b81aaf429985b108e3a9f92f6e [sys.catalog]: This master's current role is: LEADER
I20250903 08:08:56.442752 22544 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3081f8b81aaf429985b108e3a9f92f6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "3081f8b81aaf429985b108e3a9f92f6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3081f8b81aaf429985b108e3a9f92f6e" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 42385 } } }
I20250903 08:08:56.442839 22544 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3081f8b81aaf429985b108e3a9f92f6e [sys.catalog]: This master's current role is: LEADER
I20250903 08:08:56.443712 22552 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250903 08:08:56.446897 22552 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250903 08:08:56.447590 22552 catalog_manager.cc:1261] Loaded cluster ID: 8735117e5924418bbe93e94191545355
I20250903 08:08:56.447654 22552 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250903 08:08:56.450047 22552 catalog_manager.cc:1506] Loading token signing keys...
I20250903 08:08:56.450719 22552 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 3081f8b81aaf429985b108e3a9f92f6e: Loaded TSK: 0
I20250903 08:08:56.451967 22552 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250903 08:08:56.518370 22541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:08:56.518622 22541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:08:56.518715 22541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:08:56.521970 22541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 08:08:56.522045 22541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:08:56.522163 22541 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.201.193
I20250903 08:08:56.526484 22541 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.201.212:41737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/krb5kdc/oryx_127.12.201.193.keytab
--principal=oryx/127.12.201.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.201.193:46833
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.201.193
--webserver_port=38515
--webserver_require_spnego=true
--tserver_master_addrs=127.12.201.254:42385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22541
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.201.193
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 07:50:08 UTC on 5fd53c4cbb9d
build id 7874
I20250903 08:08:56.527563 22541 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:08:56.528407 22541 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:08:56.534647 22568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:56.534649 22566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:56.534649 22565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:08:56.534957 22541 server_base.cc:1047] running on GCE node
I20250903 08:08:56.535498 22541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:08:56.536041 22541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:08:56.537204 22541 hybrid_clock.cc:648] HybridClock initialized: now 1756886936537181 us; error 41 us; skew 500 ppm
Sep 03 08:08:56 dist-test-slave-27r9 krb5kdc[21949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886936, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.201.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 08:08:56.539799 22541 init.cc:377] Logged in from keytab as oryx/127.12.201.193@KRBTEST.COM (short username oryx)
I20250903 08:08:56.540838 22541 webserver.cc:480] Webserver started at http://127.12.201.193:38515/ using document root <none> and password file <none>
I20250903 08:08:56.541371 22541 fs_manager.cc:362] Metadata directory not provided
I20250903 08:08:56.541431 22541 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:08:56.544864 22541 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20250903 08:08:56.546725 22575 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:56.547808 22541 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20250903 08:08:56.547940 22541 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-0/wal
uuid: "efa9f2d3f2274aefbe3b8b25325ef7eb"
format_stamp: "Formatted at 2025-09-03 08:08:55 on dist-test-slave-27r9"
server_key: "136f66818a490118f9bf614ebd08eaaf"
server_key_iv: "9205f73ad3ac1998a8f505b981ee8c2b"
server_key_version: "encryptionkey@0"
I20250903 08:08:56.548314 22541 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:08:56.562589 22541 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:08:56.565543 22541 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:08:56.565733 22541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:08:56.566308 22541 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 08:08:56.567206 22541 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 08:08:56.567258 22541 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:56.567317 22541 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 08:08:56.567344 22541 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:56.578032 22541 rpc_server.cc:307] RPC server started. Bound to: 127.12.201.193:46833
I20250903 08:08:56.578069 22688 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.201.193:46833 every 8 connection(s)
I20250903 08:08:56.579283 22541 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-0/data/info.pb
I20250903 08:08:56.583765 13095 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUML90r/build/debug/bin/kudu as pid 22541
I20250903 08:08:56.584280 13095 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUML90r/build/debug/bin/kudu
/tmp/dist-test-taskUML90r/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.201.194:41023
--local_ip_for_outbound_sockets=127.12.201.194
--tserver_master_addrs=127.12.201.254:42385
--webserver_port=38001
--webserver_interface=127.12.201.194
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.201.212:41737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.12.201.194
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/krb5kdc/oryx_127.12.201.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 03 08:08:56 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.193@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
Sep 03 08:08:56 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.193@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
W20250903 08:08:56.592334 22692 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 08:08:56.581566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.193:35665 (local address 127.12.201.254:42385)
0903 08:08:56.582165 (+ 599us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:56.582177 (+ 12us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:56.582224 (+ 47us) server_negotiation.cc:408] Connection header received
0903 08:08:56.583199 (+ 975us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:56.583215 (+ 16us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:56.583576 (+ 361us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:56.583938 (+ 362us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:56.585618 (+ 1680us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:56.586957 (+ 1339us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:56.587967 (+ 1010us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:56.588235 (+ 268us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:56.591752 (+ 3517us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.201.193:35665: BlockingRecv error: recv got EOF from 127.12.201.193:35665 (error 108)
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":135,"threads_started":1}
I20250903 08:08:56.592331 22691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:56.581536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:42385 (local address 127.12.201.193:35665)
0903 08:08:56.581976 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:56.582020 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:56.582763 (+ 743us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:56.584061 (+ 1298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:56.584071 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:56.584629 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:08:56.585373 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:56.585394 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:56.587153 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:56.587158 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:56.587829 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:56.587842 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:56.588063 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:56.589262 (+ 1199us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:08:56.589288 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:08:56.591537 (+ 2249us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.201.254:42385: Server oryx/127.12.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":100,"threads_started":1}
W20250903 08:08:56.592960 22689 heartbeater.cc:646] Failed to heartbeat to 127.12.201.254:42385 (0 consecutive failures): Not authorized: Failed to ping master at 127.12.201.254:42385: Client connection negotiation failed: client connection to 127.12.201.254:42385: Server oryx/127.12.201.254@KRBTEST.COM not found in Kerberos database
Sep 03 08:08:56 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.193@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
Sep 03 08:08:56 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.193@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
I20250903 08:08:56.597980 22691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:56.593597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:42385 (local address 127.12.201.193:38419)
0903 08:08:56.593757 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:56.593768 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:56.593858 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:56.594081 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:56.594084 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:56.594247 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:08:56.594429 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:56.594434 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:56.595182 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:56.595185 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:56.595544 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:56.595550 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:56.595652 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:56.596205 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:08:56.596218 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:08:56.597840 (+ 1622us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.201.254:42385: Server oryx/127.12.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":91}
W20250903 08:08:56.598088 22692 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 08:08:56.593506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.193:38419 (local address 127.12.201.254:42385)
0903 08:08:56.593672 (+ 166us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:56.593676 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:56.593828 (+ 152us) server_negotiation.cc:408] Connection header received
0903 08:08:56.593940 (+ 112us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:56.593946 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:56.593987 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:56.594057 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:56.594538 (+ 481us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:56.595076 (+ 538us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:56.595644 (+ 568us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:56.595848 (+ 204us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:56.597967 (+ 2119us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.201.193:38419: BlockingRecv error: recv got EOF from 127.12.201.193:38419 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Sep 03 08:08:56 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.193@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
Sep 03 08:08:56 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.193@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
I20250903 08:08:56.603072 22691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:56.598759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:42385 (local address 127.12.201.193:37421)
0903 08:08:56.598941 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:56.598955 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:56.599068 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:56.599294 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:56.599297 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:56.599438 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:08:56.599582 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:56.599587 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:56.600311 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:56.600314 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:56.600649 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:56.600655 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:56.600753 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:56.601345 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:08:56.601357 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:08:56.602891 (+ 1534us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.201.254:42385: Server oryx/127.12.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110}
W20250903 08:08:56.603168 22692 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 08:08:56.598825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.193:37421 (local address 127.12.201.254:42385)
0903 08:08:56.598941 (+ 116us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:56.598947 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:56.599014 (+ 67us) server_negotiation.cc:408] Connection header received
0903 08:08:56.599153 (+ 139us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:56.599156 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:56.599198 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:56.599265 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:56.599687 (+ 422us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:56.600207 (+ 520us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:56.600748 (+ 541us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:56.600949 (+ 201us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:56.603051 (+ 2102us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.201.193:37421: BlockingRecv error: recv got EOF from 127.12.201.193:37421 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20250903 08:08:56.603513 22689 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20250903 08:08:56.705771 22693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:08:56.705996 22693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:08:56.706054 22693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:08:56.709225 22693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 08:08:56.709295 22693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:08:56.709424 22693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.201.194
I20250903 08:08:56.713615 22693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.201.212:41737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/krb5kdc/oryx_127.12.201.194.keytab
--principal=oryx/127.12.201.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.201.194:41023
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.201.194
--webserver_port=38001
--webserver_require_spnego=true
--tserver_master_addrs=127.12.201.254:42385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.201.194
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 07:50:08 UTC on 5fd53c4cbb9d
build id 7874
I20250903 08:08:56.714684 22693 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:08:56.715564 22693 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:08:56.721742 22701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:56.721760 22699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:56.721755 22698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:08:56.722157 22693 server_base.cc:1047] running on GCE node
I20250903 08:08:56.722522 22693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:08:56.723090 22693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:08:56.724272 22693 hybrid_clock.cc:648] HybridClock initialized: now 1756886936724250 us; error 34 us; skew 500 ppm
Sep 03 08:08:56 dist-test-slave-27r9 krb5kdc[21949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886936, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.201.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 08:08:56.727042 22693 init.cc:377] Logged in from keytab as oryx/127.12.201.194@KRBTEST.COM (short username oryx)
I20250903 08:08:56.728083 22693 webserver.cc:480] Webserver started at http://127.12.201.194:38001/ using document root <none> and password file <none>
I20250903 08:08:56.728641 22693 fs_manager.cc:362] Metadata directory not provided
I20250903 08:08:56.728708 22693 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:08:56.732404 22693 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20250903 08:08:56.734467 22708 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:56.735595 22693 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250903 08:08:56.735721 22693 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-1/wal
uuid: "4c5720a5742d4c58800e541283fbaffd"
format_stamp: "Formatted at 2025-09-03 08:08:55 on dist-test-slave-27r9"
server_key: "518b6f20e91954f07c2439cb4850127d"
server_key_iv: "1511f76ed720e2ecca9d0f317c481d17"
server_key_version: "encryptionkey@0"
I20250903 08:08:56.736120 22693 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:08:56.752653 22693 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:08:56.755477 22693 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:08:56.755656 22693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:08:56.756212 22693 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 08:08:56.757047 22693 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 08:08:56.757097 22693 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:56.757154 22693 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 08:08:56.757190 22693 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:56.766079 22693 rpc_server.cc:307] RPC server started. Bound to: 127.12.201.194:41023
I20250903 08:08:56.766119 22821 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.201.194:41023 every 8 connection(s)
I20250903 08:08:56.767149 22693 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-1/data/info.pb
I20250903 08:08:56.773051 13095 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUML90r/build/debug/bin/kudu as pid 22693
I20250903 08:08:56.773507 13095 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUML90r/build/debug/bin/kudu
/tmp/dist-test-taskUML90r/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.201.195:41079
--local_ip_for_outbound_sockets=127.12.201.195
--tserver_master_addrs=127.12.201.254:42385
--webserver_port=38421
--webserver_interface=127.12.201.195
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.201.212:41737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.12.201.195
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/krb5kdc/oryx_127.12.201.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 03 08:08:56 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.194@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
Sep 03 08:08:56 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.194@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
W20250903 08:08:56.776970 22692 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 08:08:56.768868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.194:53901 (local address 127.12.201.254:42385)
0903 08:08:56.769004 (+ 136us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:56.769008 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:56.769660 (+ 652us) server_negotiation.cc:408] Connection header received
0903 08:08:56.770510 (+ 850us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:56.770513 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:56.770582 (+ 69us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:56.770663 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:56.772026 (+ 1363us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:56.772534 (+ 508us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:56.773174 (+ 640us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:56.773364 (+ 190us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:56.776848 (+ 3484us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.201.194:53901: BlockingRecv error: recv got EOF from 127.12.201.194:53901 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20250903 08:08:56.777443 22824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:56.769127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:42385 (local address 127.12.201.194:53901)
0903 08:08:56.769524 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:56.769559 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:56.770305 (+ 746us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:56.770837 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:56.770846 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:56.771301 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:08:56.771875 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:56.771888 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:56.772656 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:56.772659 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:56.773057 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:56.773064 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:56.773261 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:56.774480 (+ 1219us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:08:56.774503 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:08:56.776704 (+ 2201us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.201.254:42385: Server oryx/127.12.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":100,"threads_started":1}
W20250903 08:08:56.777854 22822 heartbeater.cc:646] Failed to heartbeat to 127.12.201.254:42385 (0 consecutive failures): Not authorized: Failed to ping master at 127.12.201.254:42385: Client connection negotiation failed: client connection to 127.12.201.254:42385: Server oryx/127.12.201.254@KRBTEST.COM not found in Kerberos database
Sep 03 08:08:56 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.194@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
Sep 03 08:08:56 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.194@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
I20250903 08:08:56.782745 22824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:56.778313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:42385 (local address 127.12.201.194:34913)
0903 08:08:56.778446 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:56.778461 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:56.778543 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:56.778777 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:56.778779 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:56.778922 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:08:56.779137 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:56.779143 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:56.779825 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:56.779828 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:56.780211 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:56.780218 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:56.780323 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:56.780924 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:08:56.780936 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:08:56.782624 (+ 1688us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.201.254:42385: Server oryx/127.12.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":46}
W20250903 08:08:56.782878 22692 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 08:08:56.778379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.194:34913 (local address 127.12.201.254:42385)
0903 08:08:56.778505 (+ 126us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:56.778510 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:56.778523 (+ 13us) server_negotiation.cc:408] Connection header received
0903 08:08:56.778635 (+ 112us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:56.778638 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:56.778684 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:56.778751 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:56.779245 (+ 494us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:56.779719 (+ 474us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:56.780317 (+ 598us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:56.780519 (+ 202us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:56.782763 (+ 2244us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.201.194:34913: BlockingRecv error: recv got EOF from 127.12.201.194:34913 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 03 08:08:56 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.194@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
Sep 03 08:08:56 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.194@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
I20250903 08:08:56.787846 22824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:56.783666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:42385 (local address 127.12.201.194:38087)
0903 08:08:56.783791 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:56.783807 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:56.783886 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:56.784136 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:56.784139 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:56.784279 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:08:56.784422 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:56.784427 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:56.785088 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:56.785093 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:56.785435 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:56.785442 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:56.785547 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:56.786082 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:08:56.786097 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:08:56.787714 (+ 1617us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.201.254:42385: Server oryx/127.12.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":42}
W20250903 08:08:56.787946 22692 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 08:08:56.783741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.194:38087 (local address 127.12.201.254:42385)
0903 08:08:56.783864 (+ 123us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:56.783868 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:56.783880 (+ 12us) server_negotiation.cc:408] Connection header received
0903 08:08:56.784001 (+ 121us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:56.784004 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:56.784044 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:56.784112 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:56.784527 (+ 415us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:56.784986 (+ 459us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:56.785545 (+ 559us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:56.785750 (+ 205us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:56.787831 (+ 2081us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.201.194:38087: BlockingRecv error: recv got EOF from 127.12.201.194:38087 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20250903 08:08:56.788203 22822 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20250903 08:08:56.879727 22825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:08:56.879966 22825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:08:56.880023 22825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:08:56.883306 22825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 08:08:56.883378 22825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:08:56.883458 22825 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.201.195
I20250903 08:08:56.887713 22825 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.201.212:41737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/krb5kdc/oryx_127.12.201.195.keytab
--principal=oryx/127.12.201.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.201.195:41079
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.201.195
--webserver_port=38421
--webserver_require_spnego=true
--tserver_master_addrs=127.12.201.254:42385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22825
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.201.195
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 07:50:08 UTC on 5fd53c4cbb9d
build id 7874
I20250903 08:08:56.888842 22825 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:08:56.889674 22825 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:08:56.896078 22833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:56.896153 22831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:08:56.896072 22830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:08:56.896366 22825 server_base.cc:1047] running on GCE node
I20250903 08:08:56.896821 22825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:08:56.897392 22825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:08:56.898559 22825 hybrid_clock.cc:648] HybridClock initialized: now 1756886936898542 us; error 30 us; skew 500 ppm
Sep 03 08:08:56 dist-test-slave-27r9 krb5kdc[21949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886936, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.201.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 08:08:56.901229 22825 init.cc:377] Logged in from keytab as oryx/127.12.201.195@KRBTEST.COM (short username oryx)
I20250903 08:08:56.902192 22825 webserver.cc:480] Webserver started at http://127.12.201.195:38421/ using document root <none> and password file <none>
I20250903 08:08:56.902693 22825 fs_manager.cc:362] Metadata directory not provided
I20250903 08:08:56.902748 22825 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:08:56.906180 22825 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20250903 08:08:56.907982 22840 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:56.908929 22825 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250903 08:08:56.909032 22825 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-2/wal
uuid: "fc2c501335b34acb8f4ef6df0accf5c8"
format_stamp: "Formatted at 2025-09-03 08:08:56 on dist-test-slave-27r9"
server_key: "962d79e7297d9f14b9bf5d7e802f1560"
server_key_iv: "a18547766b4812a9cf6bb54692576a80"
server_key_version: "encryptionkey@0"
I20250903 08:08:56.909425 22825 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:08:56.927170 22825 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:08:56.929980 22825 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:08:56.930148 22825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:08:56.930667 22825 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 08:08:56.931525 22825 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 08:08:56.931574 22825 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:56.931613 22825 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 08:08:56.931631 22825 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 08:08:56.940605 22825 rpc_server.cc:307] RPC server started. Bound to: 127.12.201.195:41079
I20250903 08:08:56.940644 22953 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.201.195:41079 every 8 connection(s)
I20250903 08:08:56.941627 22825 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756886860253814-13095-0/minicluster-data/ts-2/data/info.pb
Sep 03 08:08:56 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.195@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
I20250903 08:08:56.949559 13095 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUML90r/build/debug/bin/kudu as pid 22825
Sep 03 08:08:56 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.195@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
W20250903 08:08:56.951390 22692 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 08:08:56.943323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.195:49581 (local address 127.12.201.254:42385)
0903 08:08:56.943508 (+ 185us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:56.943513 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:56.944182 (+ 669us) server_negotiation.cc:408] Connection header received
0903 08:08:56.945003 (+ 821us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:56.945007 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:56.945061 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:56.945142 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:56.946394 (+ 1252us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:56.946873 (+ 479us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:56.947538 (+ 665us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:56.947713 (+ 175us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:56.951270 (+ 3557us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.201.195:49581: BlockingRecv error: recv got EOF from 127.12.201.195:49581 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20250903 08:08:56.951880 22956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:56.943618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:42385 (local address 127.12.201.195:49581)
0903 08:08:56.944052 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:56.944086 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:56.944817 (+ 731us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:56.945307 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:56.945314 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:56.945710 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:08:56.946252 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:56.946263 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:56.947020 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:56.947024 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:56.947422 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:56.947429 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:56.947622 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:56.948707 (+ 1085us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:08:56.948729 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:08:56.951154 (+ 2425us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.201.254:42385: Server oryx/127.12.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":124,"threads_started":1}
W20250903 08:08:56.952315 22954 heartbeater.cc:646] Failed to heartbeat to 127.12.201.254:42385 (0 consecutive failures): Not authorized: Failed to ping master at 127.12.201.254:42385: Client connection negotiation failed: client connection to 127.12.201.254:42385: Server oryx/127.12.201.254@KRBTEST.COM not found in Kerberos database
Sep 03 08:08:56 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.195@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
Sep 03 08:08:56 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.195@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
I20250903 08:08:56.957882 22956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:56.952760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:42385 (local address 127.12.201.195:45081)
0903 08:08:56.952864 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:56.952878 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:56.952949 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:56.953166 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:56.953169 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:56.953357 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:08:56.953603 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:56.953612 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:56.954775 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:56.954778 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:56.955282 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:56.955290 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:56.955397 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:56.955851 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:08:56.955864 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:08:56.957784 (+ 1920us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.201.254:42385: Server oryx/127.12.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":27}
W20250903 08:08:56.958060 22692 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 08:08:56.952814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.195:45081 (local address 127.12.201.254:42385)
0903 08:08:56.952913 (+ 99us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:56.952916 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:56.952930 (+ 14us) server_negotiation.cc:408] Connection header received
0903 08:08:56.953035 (+ 105us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:56.953037 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:56.953071 (+ 34us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:56.953134 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:56.953750 (+ 616us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:56.954594 (+ 844us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:56.955394 (+ 800us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:56.955589 (+ 195us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:56.957926 (+ 2337us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.201.195:45081: BlockingRecv error: recv got EOF from 127.12.201.195:45081 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20250903 08:08:56.959743 22958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:56.950462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54740 (local address 127.12.201.254:42385)
0903 08:08:56.950693 (+ 231us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:56.950696 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:56.950706 (+ 10us) server_negotiation.cc:408] Connection header received
0903 08:08:56.950735 (+ 29us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:56.950738 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:56.950773 (+ 35us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:56.950854 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:56.951772 (+ 918us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:56.952513 (+ 741us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:56.953332 (+ 819us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:56.953462 (+ 130us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:56.955045 (+ 1583us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:08:56.955066 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:08:56.955079 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:08:56.955118 (+ 39us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:08:56.957241 (+ 2123us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:56.957800 (+ 559us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:56.957806 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:56.957812 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:56.957918 (+ 106us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:08:56.958519 (+ 601us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:08:56.958524 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:08:56.958526 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:08:56.958853 (+ 327us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:08:56.959074 (+ 221us) server_negotiation.cc:1036] Waiting for connection context
0903 08:08:56.959408 (+ 334us) server_negotiation.cc:300] Negotiation successful
0903 08:08:56.959628 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":82,"threads_started":1}
Sep 03 08:08:56 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.195@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
Sep 03 08:08:56 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.195@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
I20250903 08:08:56.963502 22956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:56.958497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:42385 (local address 127.12.201.195:34203)
0903 08:08:56.958746 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:56.958761 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:56.958841 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:56.959085 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:56.959088 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:56.959246 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:08:56.959390 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:56.959396 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:56.960311 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:56.960314 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:56.960698 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:56.960707 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:56.960820 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:56.961306 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:08:56.961320 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:08:56.963365 (+ 2045us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.201.254:42385: Server oryx/127.12.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183}
W20250903 08:08:56.963585 22692 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 08:08:56.958615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.195:34203 (local address 127.12.201.254:42385)
0903 08:08:56.958739 (+ 124us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:56.958743 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:56.958801 (+ 58us) server_negotiation.cc:408] Connection header received
0903 08:08:56.958928 (+ 127us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:56.958931 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:56.958972 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:56.959073 (+ 101us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:56.959602 (+ 529us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:56.960191 (+ 589us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:56.960811 (+ 620us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:56.960936 (+ 125us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:56.963481 (+ 2545us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.201.195:34203: BlockingRecv error: recv got EOF from 127.12.201.195:34203 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20250903 08:08:56.963824 22954 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 03 08:08:57 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.193@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
Sep 03 08:08:57 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.193@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
I20250903 08:08:57.609252 22959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:57.604186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:42385 (local address 127.12.201.193:34775)
0903 08:08:57.604548 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:57.604561 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:57.604683 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:57.605022 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:57.605025 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:57.605263 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:08:57.605463 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:57.605469 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:57.606302 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:57.606306 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:57.606710 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:57.606717 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:57.606818 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:57.607450 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:08:57.607463 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:08:57.609134 (+ 1671us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.201.254:42385: Server oryx/127.12.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":88,"threads_started":1}
W20250903 08:08:57.609416 22960 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 08:08:57.604285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.193:34775 (local address 127.12.201.254:42385)
0903 08:08:57.604552 (+ 267us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:57.604557 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:57.604662 (+ 105us) server_negotiation.cc:408] Connection header received
0903 08:08:57.604801 (+ 139us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:57.604804 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:57.604862 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:57.604954 (+ 92us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:57.605569 (+ 615us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:57.606182 (+ 613us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:57.606813 (+ 631us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:57.606947 (+ 134us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:57.609259 (+ 2312us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.201.193:34775: BlockingRecv error: recv got EOF from 127.12.201.193:34775 (error 108)
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":63,"threads_started":1}
Sep 03 08:08:57 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.194@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
Sep 03 08:08:57 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.194@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
I20250903 08:08:57.793926 22961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:57.788883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:42385 (local address 127.12.201.194:32839)
0903 08:08:57.789172 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:57.789192 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:57.789336 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:57.789621 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:57.789628 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:57.789828 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:08:57.790076 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:57.790084 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:57.790845 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:57.790849 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:57.791318 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:57.791327 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:57.791437 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:57.792012 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:08:57.792025 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:08:57.793762 (+ 1737us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.201.254:42385: Server oryx/127.12.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":80,"threads_started":1}
W20250903 08:08:57.793990 22960 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 08:08:57.789006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.194:32839 (local address 127.12.201.254:42385)
0903 08:08:57.789160 (+ 154us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:57.789166 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:57.789279 (+ 113us) server_negotiation.cc:408] Connection header received
0903 08:08:57.789440 (+ 161us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:57.789446 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:57.789502 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:57.789591 (+ 89us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:57.790201 (+ 610us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:57.790737 (+ 536us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:57.791435 (+ 698us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:57.791641 (+ 206us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:57.793887 (+ 2246us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.201.194:32839: BlockingRecv error: recv got EOF from 127.12.201.194:32839 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 03 08:08:57 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.195@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
Sep 03 08:08:57 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.195@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
I20250903 08:08:57.968976 22962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:57.964418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:42385 (local address 127.12.201.195:59217)
0903 08:08:57.964671 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:57.964684 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:57.964783 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:57.965001 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:57.965003 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:57.965185 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:08:57.965395 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:57.965401 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:57.966049 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:57.966052 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:57.966445 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:57.966451 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:57.966533 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:57.967149 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:08:57.967161 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:08:57.968863 (+ 1702us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.201.254:42385: Server oryx/127.12.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":76,"threads_started":1}
W20250903 08:08:57.969151 22960 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 08:08:57.964504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.195:59217 (local address 127.12.201.254:42385)
0903 08:08:57.964657 (+ 153us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:57.964661 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:57.964748 (+ 87us) server_negotiation.cc:408] Connection header received
0903 08:08:57.964860 (+ 112us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:57.964863 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:57.964912 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:57.964981 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:57.965487 (+ 506us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:57.965964 (+ 477us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:57.966570 (+ 606us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:57.966747 (+ 177us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:57.969014 (+ 2267us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.201.195:59217: BlockingRecv error: recv got EOF from 127.12.201.195:59217 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 03 08:08:58 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.193@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
Sep 03 08:08:58 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.193@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
I20250903 08:08:58.621709 22963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:58.609890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:42385 (local address 127.12.201.193:54581)
0903 08:08:58.610163 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:58.610181 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:58.610280 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:58.610566 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:58.610569 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:58.610763 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:08:58.610964 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:58.610971 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:58.611755 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:58.611758 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:58.612140 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:58.612147 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:58.612238 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:58.619534 (+ 7296us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:08:58.619553 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:08:58.621544 (+ 1991us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.201.254:42385: Server oryx/127.12.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":73,"threads_started":1}
W20250903 08:08:58.621819 22964 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 08:08:58.609998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.193:54581 (local address 127.12.201.254:42385)
0903 08:08:58.610250 (+ 252us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:58.610254 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:58.610267 (+ 13us) server_negotiation.cc:408] Connection header received
0903 08:08:58.610370 (+ 103us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:58.610373 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:58.610423 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:58.610500 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:58.611130 (+ 630us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:58.611659 (+ 529us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:58.612224 (+ 565us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:58.612352 (+ 128us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:58.621692 (+ 9340us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.201.193:54581: BlockingRecv error: recv got EOF from 127.12.201.193:54581 (error 108)
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":87,"threads_started":1}
Sep 03 08:08:58 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.194@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
Sep 03 08:08:58 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.194@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
I20250903 08:08:58.799455 22965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:58.794616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:42385 (local address 127.12.201.194:59051)
0903 08:08:58.794916 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:58.794928 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:58.795059 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:58.795328 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:58.795331 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:58.795527 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:08:58.795715 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:58.795722 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:58.796428 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:58.796434 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:58.796803 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:58.796810 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:58.796915 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:58.797483 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:08:58.797495 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:08:58.799290 (+ 1795us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.201.254:42385: Server oryx/127.12.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":100,"threads_started":1}
W20250903 08:08:58.799541 22964 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 08:08:58.794715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.194:59051 (local address 127.12.201.254:42385)
0903 08:08:58.794856 (+ 141us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:58.794862 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:58.794995 (+ 133us) server_negotiation.cc:408] Connection header received
0903 08:08:58.795162 (+ 167us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:58.795166 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:58.795221 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:58.795290 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:58.795836 (+ 546us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:58.796317 (+ 481us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:58.796911 (+ 594us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:58.797137 (+ 226us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:58.799428 (+ 2291us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.201.194:59051: BlockingRecv error: recv got EOF from 127.12.201.194:59051 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 03 08:08:58 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.195@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
Sep 03 08:08:58 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.195@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
I20250903 08:08:58.974129 22966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:58.969737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:42385 (local address 127.12.201.195:34179)
0903 08:08:58.969990 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:58.970003 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:58.970098 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:58.970308 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:58.970310 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:58.970481 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:08:58.970672 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:58.970677 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:58.971389 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:58.971392 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:58.971757 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:58.971764 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:58.971852 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:58.972393 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:08:58.972406 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:08:58.974012 (+ 1606us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.201.254:42385: Server oryx/127.12.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":73,"threads_started":1}
W20250903 08:08:58.974251 22964 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 08:08:58.969866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.195:34179 (local address 127.12.201.254:42385)
0903 08:08:58.969984 (+ 118us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:58.969988 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:58.970061 (+ 73us) server_negotiation.cc:408] Connection header received
0903 08:08:58.970168 (+ 107us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:58.970171 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:58.970216 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:58.970285 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:58.970757 (+ 472us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:58.971295 (+ 538us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:58.971851 (+ 556us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:58.971976 (+ 125us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:58.974127 (+ 2151us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.201.195:34179: BlockingRecv error: recv got EOF from 127.12.201.195:34179 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 03 08:08:59 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.193@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
Sep 03 08:08:59 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.193@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
I20250903 08:08:59.627393 22967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:59.622514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:42385 (local address 127.12.201.193:59807)
0903 08:08:59.622810 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:59.622825 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:59.622926 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:59.623203 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:59.623205 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:59.623388 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:08:59.623624 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:59.623633 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:59.624484 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:59.624488 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:59.624987 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:59.624995 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:59.625087 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:59.625629 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:08:59.625643 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:08:59.627276 (+ 1633us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.201.254:42385: Server oryx/127.12.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":97,"threads_started":1}
W20250903 08:08:59.627517 22968 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 08:08:59.622652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.193:59807 (local address 127.12.201.254:42385)
0903 08:08:59.622876 (+ 224us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:59.622879 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:59.622893 (+ 14us) server_negotiation.cc:408] Connection header received
0903 08:08:59.623026 (+ 133us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:59.623032 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:59.623091 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:59.623169 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:59.623765 (+ 596us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:59.624368 (+ 603us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:59.625091 (+ 723us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:59.625239 (+ 148us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:59.627403 (+ 2164us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.201.193:59807: BlockingRecv error: recv got EOF from 127.12.201.193:59807 (error 108)
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":61,"threads_started":1}
Sep 03 08:08:59 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.194@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
Sep 03 08:08:59 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.194@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
I20250903 08:08:59.805236 22969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:59.800265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:42385 (local address 127.12.201.194:42629)
0903 08:08:59.800636 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:59.800652 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:59.800772 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:59.801026 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:59.801030 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:59.801239 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:08:59.801428 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:59.801434 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:59.802163 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:59.802167 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:59.802552 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:59.802559 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:59.802667 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:59.803327 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:08:59.803341 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:08:59.805092 (+ 1751us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.201.254:42385: Server oryx/127.12.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":99,"threads_started":1}
W20250903 08:08:59.805419 22968 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 08:08:59.800385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.194:42629 (local address 127.12.201.254:42385)
0903 08:08:59.800600 (+ 215us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:59.800605 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:59.800728 (+ 123us) server_negotiation.cc:408] Connection header received
0903 08:08:59.800867 (+ 139us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:59.800871 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:59.800924 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:59.800992 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:59.801567 (+ 575us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:59.802053 (+ 486us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:59.802660 (+ 607us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:59.802872 (+ 212us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:59.805279 (+ 2407us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.201.194:42629: BlockingRecv error: recv got EOF from 127.12.201.194:42629 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
Sep 03 08:08:59 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.195@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
Sep 03 08:08:59 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.195@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
I20250903 08:08:59.979334 22970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:08:59.974877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:42385 (local address 127.12.201.195:52587)
0903 08:08:59.975151 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0903 08:08:59.975165 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0903 08:08:59.975289 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:08:59.975512 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:08:59.975515 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:08:59.975689 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:08:59.975880 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:59.975885 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:59.976535 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:08:59.976537 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:08:59.976903 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:08:59.976909 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:08:59.976992 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:59.977518 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:08:59.977529 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:08:59.979220 (+ 1691us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.201.254:42385: Server oryx/127.12.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":98,"threads_started":1}
W20250903 08:08:59.979476 22968 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 08:08:59.974956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.195:52587 (local address 127.12.201.254:42385)
0903 08:08:59.975154 (+ 198us) server_negotiation.cc:207] Beginning negotiation
0903 08:08:59.975158 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:08:59.975248 (+ 90us) server_negotiation.cc:408] Connection header received
0903 08:08:59.975366 (+ 118us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:08:59.975369 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:08:59.975414 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:08:59.975480 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:08:59.975975 (+ 495us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:59.976450 (+ 475us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:08:59.976988 (+ 538us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:08:59.977111 (+ 123us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:08:59.979342 (+ 2231us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.201.195:52587: BlockingRecv error: recv got EOF from 127.12.201.195:52587 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 03 08:09:00 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.193@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
Sep 03 08:09:00 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.193@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
I20250903 08:09:00.632771 22971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:09:00.628099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:42385 (local address 127.12.201.193:44105)
0903 08:09:00.628409 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0903 08:09:00.628423 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0903 08:09:00.628518 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:09:00.628768 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:09:00.628770 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:09:00.628955 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:09:00.629150 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:09:00.629157 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:09:00.629830 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:09:00.629833 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:09:00.630203 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:09:00.630210 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:09:00.630303 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:00.630817 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:09:00.630836 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:09:00.632654 (+ 1818us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.201.254:42385: Server oryx/127.12.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":86,"threads_started":1}
W20250903 08:09:00.632931 22972 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 08:09:00.628209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.193:44105 (local address 127.12.201.254:42385)
0903 08:09:00.628442 (+ 233us) server_negotiation.cc:207] Beginning negotiation
0903 08:09:00.628445 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 08:09:00.628477 (+ 32us) server_negotiation.cc:408] Connection header received
0903 08:09:00.628614 (+ 137us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:09:00.628617 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:09:00.628661 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:09:00.628736 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:09:00.629245 (+ 509us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:00.629748 (+ 503us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:09:00.630290 (+ 542us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:00.630420 (+ 130us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:00.632784 (+ 2364us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.201.193:44105: BlockingRecv error: recv got EOF from 127.12.201.193:44105 (error 108)
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":61,"threads_started":1}
Sep 03 08:09:00 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.194@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
Sep 03 08:09:00 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.194@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
I20250903 08:09:00.810545 22973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:09:00.806049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:42385 (local address 127.12.201.194:41843)
0903 08:09:00.806294 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0903 08:09:00.806307 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0903 08:09:00.806417 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:09:00.806639 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:09:00.806641 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:09:00.806816 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:09:00.807061 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:09:00.807069 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:09:00.807747 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:09:00.807749 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:09:00.808112 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:09:00.808118 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:09:00.808203 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:00.808740 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:09:00.808752 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:09:00.810409 (+ 1657us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.201.254:42385: Server oryx/127.12.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":71,"threads_started":1}
W20250903 08:09:00.810681 22972 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 08:09:00.806167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.194:41843 (local address 127.12.201.254:42385)
0903 08:09:00.806293 (+ 126us) server_negotiation.cc:207] Beginning negotiation
0903 08:09:00.806297 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:09:00.806392 (+ 95us) server_negotiation.cc:408] Connection header received
0903 08:09:00.806493 (+ 101us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:09:00.806495 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:09:00.806540 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:09:00.806615 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:09:00.807156 (+ 541us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:00.807655 (+ 499us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:09:00.808191 (+ 536us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:00.808333 (+ 142us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:00.810543 (+ 2210us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.201.194:41843: BlockingRecv error: recv got EOF from 127.12.201.194:41843 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 03 08:09:00 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.195@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
Sep 03 08:09:00 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.195@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
I20250903 08:09:00.984894 22974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:09:00.980051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:42385 (local address 127.12.201.195:35713)
0903 08:09:00.980346 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0903 08:09:00.980360 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0903 08:09:00.980467 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:09:00.980694 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:09:00.980696 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:09:00.980863 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:09:00.981051 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:09:00.981057 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:09:00.981743 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:09:00.981747 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:09:00.982202 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:09:00.982214 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:09:00.982346 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:00.983022 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:09:00.983038 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:09:00.984782 (+ 1744us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.201.254:42385: Server oryx/127.12.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":71,"threads_started":1}
W20250903 08:09:00.985093 22972 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 08:09:00.980234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.195:35713 (local address 127.12.201.254:42385)
0903 08:09:00.980366 (+ 132us) server_negotiation.cc:207] Beginning negotiation
0903 08:09:00.980369 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 08:09:00.980429 (+ 60us) server_negotiation.cc:408] Connection header received
0903 08:09:00.980537 (+ 108us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:09:00.980539 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:09:00.980585 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:09:00.980661 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:09:00.981134 (+ 473us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:00.981627 (+ 493us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:09:00.982317 (+ 690us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:00.982524 (+ 207us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:00.984902 (+ 2378us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.201.195:35713: BlockingRecv error: recv got EOF from 127.12.201.195:35713 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 03 08:09:01 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.193@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
Sep 03 08:09:01 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.193@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
I20250903 08:09:01.638556 22975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:09:01.633590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:42385 (local address 127.12.201.193:41861)
0903 08:09:01.633921 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0903 08:09:01.633935 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0903 08:09:01.634026 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:09:01.634264 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:09:01.634266 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:09:01.634453 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:09:01.634649 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:09:01.634655 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:09:01.635330 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:09:01.635332 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:09:01.635722 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:09:01.635728 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:09:01.635812 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:01.636363 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:09:01.636375 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:09:01.638395 (+ 2020us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.201.254:42385: Server oryx/127.12.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":84,"threads_started":1}
W20250903 08:09:01.638762 22976 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 08:09:01.633700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.193:41861 (local address 127.12.201.254:42385)
0903 08:09:01.633986 (+ 286us) server_negotiation.cc:207] Beginning negotiation
0903 08:09:01.633990 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:09:01.634003 (+ 13us) server_negotiation.cc:408] Connection header received
0903 08:09:01.634107 (+ 104us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:09:01.634109 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:09:01.634151 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:09:01.634228 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:09:01.634738 (+ 510us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:01.635256 (+ 518us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:09:01.635801 (+ 545us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:01.635924 (+ 123us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:01.638643 (+ 2719us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.201.193:41861: BlockingRecv error: recv got EOF from 127.12.201.193:41861 (error 108)
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":122,"threads_started":1}
Sep 03 08:09:01 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.194@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
Sep 03 08:09:01 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.194@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
I20250903 08:09:01.815927 22977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:09:01.811330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:42385 (local address 127.12.201.194:34699)
0903 08:09:01.811628 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0903 08:09:01.811642 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0903 08:09:01.811746 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:09:01.811995 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:09:01.811997 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:09:01.812176 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:09:01.812366 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:09:01.812372 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:09:01.813045 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:09:01.813048 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:09:01.813431 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:09:01.813441 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:09:01.813552 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:01.814102 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:09:01.814115 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:09:01.815814 (+ 1699us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.201.254:42385: Server oryx/127.12.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":104,"threads_started":1}
W20250903 08:09:01.816053 22976 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 08:09:01.811466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.194:34699 (local address 127.12.201.254:42385)
0903 08:09:01.811618 (+ 152us) server_negotiation.cc:207] Beginning negotiation
0903 08:09:01.811621 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 08:09:01.811714 (+ 93us) server_negotiation.cc:408] Connection header received
0903 08:09:01.811848 (+ 134us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:09:01.811851 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:09:01.811904 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:09:01.811971 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:09:01.812467 (+ 496us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:01.812963 (+ 496us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:09:01.813542 (+ 579us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:01.813690 (+ 148us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:01.815945 (+ 2255us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.201.194:34699: BlockingRecv error: recv got EOF from 127.12.201.194:34699 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 03 08:09:01 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.195@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
Sep 03 08:09:01 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.195@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
I20250903 08:09:01.990386 22978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:09:01.985586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:42385 (local address 127.12.201.195:45151)
0903 08:09:01.985854 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0903 08:09:01.985868 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0903 08:09:01.985958 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:09:01.986214 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:09:01.986217 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:09:01.986403 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:09:01.986603 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:09:01.986610 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:09:01.987388 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:09:01.987392 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:09:01.987799 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:09:01.987806 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:09:01.987909 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:01.988497 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:09:01.988511 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:09:01.990221 (+ 1710us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.201.254:42385: Server oryx/127.12.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":89,"threads_started":1}
W20250903 08:09:01.990453 22976 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 08:09:01.985664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.195:45151 (local address 127.12.201.254:42385)
0903 08:09:01.985822 (+ 158us) server_negotiation.cc:207] Beginning negotiation
0903 08:09:01.985826 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:09:01.985928 (+ 102us) server_negotiation.cc:408] Connection header received
0903 08:09:01.986034 (+ 106us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:09:01.986036 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:09:01.986080 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:09:01.986149 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:09:01.986726 (+ 577us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:01.987267 (+ 541us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:09:01.987901 (+ 634us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:01.988024 (+ 123us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:01.990348 (+ 2324us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.201.195:45151: BlockingRecv error: recv got EOF from 127.12.201.195:45151 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 03 08:09:02 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.193@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
Sep 03 08:09:02 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.193@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
I20250903 08:09:02.644878 22979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:09:02.639418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:42385 (local address 127.12.201.193:41245)
0903 08:09:02.639718 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0903 08:09:02.639732 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0903 08:09:02.639831 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:09:02.640089 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:09:02.640091 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:09:02.640269 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:09:02.640484 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:09:02.640491 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:09:02.641357 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:09:02.641361 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:09:02.641765 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:09:02.641773 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:09:02.641875 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:02.642586 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:09:02.642600 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:09:02.644762 (+ 2162us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.201.254:42385: Server oryx/127.12.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":91,"threads_started":1}
W20250903 08:09:02.645005 22980 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 08:09:02.639623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.193:41245 (local address 127.12.201.254:42385)
0903 08:09:02.639866 (+ 243us) server_negotiation.cc:207] Beginning negotiation
0903 08:09:02.639870 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:09:02.639883 (+ 13us) server_negotiation.cc:408] Connection header received
0903 08:09:02.639921 (+ 38us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:09:02.639924 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:09:02.639967 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:09:02.640065 (+ 98us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:09:02.640627 (+ 562us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:02.641219 (+ 592us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:09:02.641897 (+ 678us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:02.642048 (+ 151us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:02.644889 (+ 2841us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.201.193:41245: BlockingRecv error: recv got EOF from 127.12.201.193:41245 (error 108)
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":84,"threads_started":1}
Sep 03 08:09:02 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.194@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
Sep 03 08:09:02 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.194@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
I20250903 08:09:02.822000 22981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:09:02.816794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:42385 (local address 127.12.201.194:50529)
0903 08:09:02.817133 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0903 08:09:02.817151 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0903 08:09:02.817278 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:09:02.817575 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:09:02.817578 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:09:02.817791 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:09:02.817983 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:09:02.817990 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:09:02.818740 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:09:02.818744 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:09:02.819154 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:09:02.819162 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:09:02.819272 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:02.819900 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:09:02.819914 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:09:02.821830 (+ 1916us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.201.254:42385: Server oryx/127.12.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":107,"threads_started":1}
W20250903 08:09:02.822090 22980 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 08:09:02.816931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.194:50529 (local address 127.12.201.254:42385)
0903 08:09:02.817097 (+ 166us) server_negotiation.cc:207] Beginning negotiation
0903 08:09:02.817102 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 08:09:02.817231 (+ 129us) server_negotiation.cc:408] Connection header received
0903 08:09:02.817384 (+ 153us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:09:02.817388 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:09:02.817439 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:09:02.817534 (+ 95us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:09:02.818111 (+ 577us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:02.818619 (+ 508us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:09:02.819271 (+ 652us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:02.819506 (+ 235us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:02.821972 (+ 2466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.201.194:50529: BlockingRecv error: recv got EOF from 127.12.201.194:50529 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 03 08:09:02 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.195@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
Sep 03 08:09:02 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.195@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
I20250903 08:09:02.996191 22982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:09:02.991258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:42385 (local address 127.12.201.195:49783)
0903 08:09:02.991520 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0903 08:09:02.991537 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0903 08:09:02.991649 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:09:02.991880 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:09:02.991883 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:09:02.992057 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:09:02.992250 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:09:02.992256 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:09:02.992998 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:09:02.993002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:09:02.993385 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:09:02.993392 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:09:02.993495 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:02.994069 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:09:02.994082 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:09:02.996071 (+ 1989us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.201.254:42385: Server oryx/127.12.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":84,"threads_started":1}
W20250903 08:09:02.996330 22980 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 08:09:02.991394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.195:49783 (local address 127.12.201.254:42385)
0903 08:09:02.991530 (+ 136us) server_negotiation.cc:207] Beginning negotiation
0903 08:09:02.991534 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:09:02.991663 (+ 129us) server_negotiation.cc:408] Connection header received
0903 08:09:02.991714 (+ 51us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:09:02.991717 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:09:02.991767 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:09:02.991846 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:09:02.992365 (+ 519us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:02.992871 (+ 506us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:09:02.993529 (+ 658us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:02.993688 (+ 159us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:02.996213 (+ 2525us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.201.195:49783: BlockingRecv error: recv got EOF from 127.12.201.195:49783 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 03 08:09:03 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.193@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
Sep 03 08:09:03 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.193@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
I20250903 08:09:03.655288 22983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:09:03.645701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:42385 (local address 127.12.201.193:56557)
0903 08:09:03.646044 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0903 08:09:03.646058 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0903 08:09:03.646158 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:09:03.646388 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:09:03.646390 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:09:03.646575 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:09:03.646773 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:09:03.646780 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:09:03.647546 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:09:03.647549 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:09:03.647933 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:09:03.647940 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:09:03.648034 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:03.653032 (+ 4998us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:09:03.653048 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:09:03.655124 (+ 2076us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.201.254:42385: Server oryx/127.12.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":126,"threads_started":1}
W20250903 08:09:03.655398 22984 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 08:09:03.645796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.193:56557 (local address 127.12.201.254:42385)
0903 08:09:03.646056 (+ 260us) server_negotiation.cc:207] Beginning negotiation
0903 08:09:03.646059 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 08:09:03.646145 (+ 86us) server_negotiation.cc:408] Connection header received
0903 08:09:03.646236 (+ 91us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:09:03.646239 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:09:03.646283 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:09:03.646364 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:09:03.646870 (+ 506us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:03.647420 (+ 550us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:09:03.648035 (+ 615us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:03.648169 (+ 134us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:03.655267 (+ 7098us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.201.193:56557: BlockingRecv error: recv got EOF from 127.12.201.193:56557 (error 108)
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":93,"threads_started":1}
Sep 03 08:09:03 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.194@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
Sep 03 08:09:03 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.194@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
I20250903 08:09:03.827926 22985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:09:03.822792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:42385 (local address 127.12.201.194:48349)
0903 08:09:03.823148 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0903 08:09:03.823164 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0903 08:09:03.823294 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:09:03.823547 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:09:03.823551 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:09:03.823757 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:09:03.823963 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:09:03.823971 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:09:03.824703 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:09:03.824707 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:09:03.825087 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:09:03.825095 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:09:03.825203 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:03.825815 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:09:03.825830 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:09:03.827804 (+ 1974us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.201.254:42385: Server oryx/127.12.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":134,"threads_started":1}
W20250903 08:09:03.828084 22984 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 08:09:03.822941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.194:48349 (local address 127.12.201.254:42385)
0903 08:09:03.823117 (+ 176us) server_negotiation.cc:207] Beginning negotiation
0903 08:09:03.823122 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 08:09:03.823235 (+ 113us) server_negotiation.cc:408] Connection header received
0903 08:09:03.823388 (+ 153us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:09:03.823391 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:09:03.823441 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:09:03.823510 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:09:03.824091 (+ 581us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:03.824594 (+ 503us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:09:03.825200 (+ 606us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:03.825413 (+ 213us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:03.827961 (+ 2548us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.201.194:48349: BlockingRecv error: recv got EOF from 127.12.201.194:48349 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 03 08:09:03 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.195@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
Sep 03 08:09:03 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.195@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
I20250903 08:09:04.002398 22986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:09:03.997054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:42385 (local address 127.12.201.195:43011)
0903 08:09:03.997397 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0903 08:09:03.997411 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0903 08:09:03.997518 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:09:03.998021 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:09:03.998025 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:09:03.998198 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:09:03.998432 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:09:03.998439 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:09:03.999259 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:09:03.999262 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:09:03.999672 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:09:03.999680 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:09:03.999781 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:04.000370 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:09:04.000384 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:09:04.002277 (+ 1893us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.201.254:42385: Server oryx/127.12.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":80,"threads_started":1}
W20250903 08:09:04.002533 22984 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 08:09:03.997083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.195:43011 (local address 127.12.201.254:42385)
0903 08:09:03.997260 (+ 177us) server_negotiation.cc:207] Beginning negotiation
0903 08:09:03.997265 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 08:09:03.997464 (+ 199us) server_negotiation.cc:408] Connection header received
0903 08:09:03.997702 (+ 238us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:09:03.997706 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:09:03.997786 (+ 80us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:09:03.997962 (+ 176us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:09:03.998566 (+ 604us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:03.999137 (+ 571us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:09:03.999781 (+ 644us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:03.999921 (+ 140us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:04.002422 (+ 2501us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.201.195:43011: BlockingRecv error: recv got EOF from 127.12.201.195:43011 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 03 08:09:04 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.193@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
Sep 03 08:09:04 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.193@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
I20250903 08:09:04.661139 22987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:09:04.656156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:42385 (local address 127.12.201.193:50887)
0903 08:09:04.656445 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0903 08:09:04.656462 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0903 08:09:04.656557 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:09:04.656853 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:09:04.656855 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:09:04.657051 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:09:04.657250 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:09:04.657256 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:09:04.658189 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:09:04.658192 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:09:04.658569 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:09:04.658575 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:09:04.658670 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:04.659290 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:09:04.659304 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:09:04.661014 (+ 1710us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.201.254:42385: Server oryx/127.12.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":72,"threads_started":1}
W20250903 08:09:04.661378 22988 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 08:09:04.656319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.193:50887 (local address 127.12.201.254:42385)
0903 08:09:04.656612 (+ 293us) server_negotiation.cc:207] Beginning negotiation
0903 08:09:04.656616 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:09:04.656629 (+ 13us) server_negotiation.cc:408] Connection header received
0903 08:09:04.656667 (+ 38us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:09:04.656679 (+ 12us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:09:04.656738 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:09:04.656820 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:09:04.657421 (+ 601us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:04.658092 (+ 671us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:09:04.658677 (+ 585us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:04.658817 (+ 140us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:04.661225 (+ 2408us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.201.193:50887: BlockingRecv error: recv got EOF from 127.12.201.193:50887 (error 108)
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":80,"threads_started":1}
Sep 03 08:09:04 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.194@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
Sep 03 08:09:04 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.194@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
I20250903 08:09:04.833832 22989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:09:04.828827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:42385 (local address 127.12.201.194:51581)
0903 08:09:04.829133 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0903 08:09:04.829149 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0903 08:09:04.829271 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:09:04.829520 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:09:04.829523 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:09:04.829722 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:09:04.829920 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:09:04.829927 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:09:04.830823 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:09:04.830829 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:09:04.831233 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:09:04.831240 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:09:04.831345 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:04.831914 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:09:04.831927 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:09:04.833676 (+ 1749us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.201.254:42385: Server oryx/127.12.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":84,"threads_started":1}
W20250903 08:09:04.833928 22988 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 08:09:04.828873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.194:51581 (local address 127.12.201.254:42385)
0903 08:09:04.829031 (+ 158us) server_negotiation.cc:207] Beginning negotiation
0903 08:09:04.829035 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:09:04.829228 (+ 193us) server_negotiation.cc:408] Connection header received
0903 08:09:04.829367 (+ 139us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:09:04.829371 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:09:04.829420 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:09:04.829490 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:09:04.830040 (+ 550us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:04.830715 (+ 675us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:09:04.831343 (+ 628us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:04.831555 (+ 212us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:04.833816 (+ 2261us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.201.194:51581: BlockingRecv error: recv got EOF from 127.12.201.194:51581 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 03 08:09:05 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.195@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
Sep 03 08:09:05 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.195@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
I20250903 08:09:05.007833 22990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:09:05.003296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:42385 (local address 127.12.201.195:44043)
0903 08:09:05.003544 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0903 08:09:05.003556 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0903 08:09:05.003650 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:09:05.003880 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:09:05.003883 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:09:05.004065 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:09:05.004269 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:09:05.004274 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:09:05.004914 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:09:05.004916 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:09:05.005303 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:09:05.005308 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:09:05.005388 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:05.005940 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:09:05.005953 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:09:05.007717 (+ 1764us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.201.254:42385: Server oryx/127.12.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":71,"threads_started":1}
W20250903 08:09:05.008041 22988 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 08:09:05.003383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.195:44043 (local address 127.12.201.254:42385)
0903 08:09:05.003506 (+ 123us) server_negotiation.cc:207] Beginning negotiation
0903 08:09:05.003509 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 08:09:05.003616 (+ 107us) server_negotiation.cc:408] Connection header received
0903 08:09:05.003722 (+ 106us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:09:05.003724 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:09:05.003770 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:09:05.003848 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:09:05.004356 (+ 508us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:05.004838 (+ 482us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:09:05.005377 (+ 539us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:05.005495 (+ 118us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:05.007913 (+ 2418us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.201.195:44043: BlockingRecv error: recv got EOF from 127.12.201.195:44043 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 03 08:09:05 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.193@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
Sep 03 08:09:05 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.193@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
I20250903 08:09:05.667136 22991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:09:05.661945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:42385 (local address 127.12.201.193:52049)
0903 08:09:05.662242 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0903 08:09:05.662256 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0903 08:09:05.662351 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:09:05.662592 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:09:05.662594 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:09:05.662777 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:09:05.663024 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:09:05.663031 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:09:05.663829 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:09:05.663832 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:09:05.664220 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:09:05.664228 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:09:05.664329 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:05.664913 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:09:05.664927 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:09:05.666933 (+ 2006us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.201.254:42385: Server oryx/127.12.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":107,"threads_started":1}
W20250903 08:09:05.667212 22992 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 08:09:05.662071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.193:52049 (local address 127.12.201.254:42385)
0903 08:09:05.662317 (+ 246us) server_negotiation.cc:207] Beginning negotiation
0903 08:09:05.662321 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:09:05.662337 (+ 16us) server_negotiation.cc:408] Connection header received
0903 08:09:05.662436 (+ 99us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:09:05.662439 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:09:05.662483 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:09:05.662561 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:09:05.663154 (+ 593us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:05.663673 (+ 519us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:09:05.664346 (+ 673us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:05.664509 (+ 163us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:05.667093 (+ 2584us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.201.193:52049: BlockingRecv error: recv got EOF from 127.12.201.193:52049 (error 108)
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":57,"threads_started":1}
Sep 03 08:09:05 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.194@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
Sep 03 08:09:05 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.194@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
I20250903 08:09:05.839870 22993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:09:05.834715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:42385 (local address 127.12.201.194:48763)
0903 08:09:05.835040 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0903 08:09:05.835058 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0903 08:09:05.835167 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:09:05.835398 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:09:05.835400 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:09:05.835582 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:09:05.835809 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:09:05.835816 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:09:05.836559 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:09:05.836563 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:09:05.836968 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:09:05.836976 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:09:05.837108 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:05.837820 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:09:05.837834 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:09:05.839745 (+ 1911us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.201.254:42385: Server oryx/127.12.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":86,"threads_started":1}
W20250903 08:09:05.840088 22992 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 08:09:05.834826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.194:48763 (local address 127.12.201.254:42385)
0903 08:09:05.834955 (+ 129us) server_negotiation.cc:207] Beginning negotiation
0903 08:09:05.834959 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:09:05.835131 (+ 172us) server_negotiation.cc:408] Connection header received
0903 08:09:05.835245 (+ 114us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:09:05.835248 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:09:05.835296 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:09:05.835371 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:09:05.835921 (+ 550us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:05.836427 (+ 506us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:09:05.837127 (+ 700us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:05.837278 (+ 151us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:05.839947 (+ 2669us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.201.194:48763: BlockingRecv error: recv got EOF from 127.12.201.194:48763 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 03 08:09:06 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.195@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
Sep 03 08:09:06 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.195@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
I20250903 08:09:06.013761 22994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:09:06.008593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:42385 (local address 127.12.201.195:53811)
0903 08:09:06.008988 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0903 08:09:06.009005 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0903 08:09:06.009146 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:09:06.009426 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:09:06.009429 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:09:06.009630 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:09:06.009824 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:09:06.009830 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:09:06.010533 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:09:06.010536 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:09:06.010907 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:09:06.010914 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:09:06.011064 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:06.011695 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:09:06.011709 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:09:06.013580 (+ 1871us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.201.254:42385: Server oryx/127.12.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":100,"threads_started":1}
W20250903 08:09:06.013828 22992 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 08:09:06.008762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.195:53811 (local address 127.12.201.254:42385)
0903 08:09:06.008935 (+ 173us) server_negotiation.cc:207] Beginning negotiation
0903 08:09:06.008940 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 08:09:06.009092 (+ 152us) server_negotiation.cc:408] Connection header received
0903 08:09:06.009261 (+ 169us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:09:06.009264 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:09:06.009310 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:09:06.009380 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:09:06.009940 (+ 560us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:06.010428 (+ 488us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:09:06.011056 (+ 628us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:06.011264 (+ 208us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:06.013718 (+ 2454us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.201.195:53811: BlockingRecv error: recv got EOF from 127.12.201.195:53811 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 03 08:09:06 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.193@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
Sep 03 08:09:06 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.193@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
I20250903 08:09:06.672757 22995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:09:06.667967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:42385 (local address 127.12.201.193:51997)
0903 08:09:06.668266 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0903 08:09:06.668281 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0903 08:09:06.668371 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:09:06.668639 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:09:06.668642 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:09:06.668826 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:09:06.669033 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:09:06.669040 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:09:06.669759 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:09:06.669762 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:09:06.670145 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:09:06.670152 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:09:06.670241 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:06.670806 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:09:06.670821 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:09:06.672638 (+ 1817us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.201.254:42385: Server oryx/127.12.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":91,"threads_started":1}
W20250903 08:09:06.672957 22996 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 08:09:06.668090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.193:51997 (local address 127.12.201.254:42385)
0903 08:09:06.668308 (+ 218us) server_negotiation.cc:207] Beginning negotiation
0903 08:09:06.668312 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:09:06.668327 (+ 15us) server_negotiation.cc:408] Connection header received
0903 08:09:06.668453 (+ 126us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:09:06.668455 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:09:06.668498 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:09:06.668576 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:09:06.669137 (+ 561us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:06.669666 (+ 529us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:09:06.670254 (+ 588us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:06.670394 (+ 140us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:06.672797 (+ 2403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.201.193:51997: BlockingRecv error: recv got EOF from 127.12.201.193:51997 (error 108)
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":56,"threads_started":1}
Sep 03 08:09:06 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.194@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
Sep 03 08:09:06 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.194@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
I20250903 08:09:06.845958 22997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:09:06.840912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:42385 (local address 127.12.201.194:35559)
0903 08:09:06.841186 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0903 08:09:06.841201 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0903 08:09:06.841302 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:09:06.841567 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:09:06.841570 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:09:06.841750 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:09:06.841955 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:09:06.841961 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:09:06.842677 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:09:06.842680 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:09:06.843145 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:09:06.843154 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:09:06.843282 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:06.843972 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:09:06.843986 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:09:06.845845 (+ 1859us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.201.254:42385: Server oryx/127.12.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":74,"threads_started":1}
W20250903 08:09:06.846123 22996 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 08:09:06.841013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.194:35559 (local address 127.12.201.254:42385)
0903 08:09:06.841173 (+ 160us) server_negotiation.cc:207] Beginning negotiation
0903 08:09:06.841178 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 08:09:06.841268 (+ 90us) server_negotiation.cc:408] Connection header received
0903 08:09:06.841380 (+ 112us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:09:06.841383 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:09:06.841426 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:09:06.841504 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:09:06.842067 (+ 563us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:06.842582 (+ 515us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:09:06.843275 (+ 693us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:06.843447 (+ 172us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:06.845976 (+ 2529us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.201.194:35559: BlockingRecv error: recv got EOF from 127.12.201.194:35559 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 03 08:09:07 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.195@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
Sep 03 08:09:07 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.195@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
I20250903 08:09:07.019559 22998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:09:07.014557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:42385 (local address 127.12.201.195:49793)
0903 08:09:07.014827 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0903 08:09:07.014844 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0903 08:09:07.014948 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:09:07.015223 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:09:07.015231 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:09:07.015436 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:09:07.015622 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:09:07.015628 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:09:07.016365 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:09:07.016368 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:09:07.016735 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:09:07.016741 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:09:07.016849 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:07.017436 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:09:07.017449 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:09:07.019394 (+ 1945us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.201.254:42385: Server oryx/127.12.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":87,"threads_started":1}
W20250903 08:09:07.019629 22996 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 08:09:07.014718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.195:49793 (local address 127.12.201.254:42385)
0903 08:09:07.014869 (+ 151us) server_negotiation.cc:207] Beginning negotiation
0903 08:09:07.014875 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0903 08:09:07.014895 (+ 20us) server_negotiation.cc:408] Connection header received
0903 08:09:07.015065 (+ 170us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:09:07.015068 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:09:07.015120 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:09:07.015190 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:09:07.015736 (+ 546us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:07.016253 (+ 517us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:09:07.016845 (+ 592us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:07.017055 (+ 210us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:07.019522 (+ 2467us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.201.195:49793: BlockingRecv error: recv got EOF from 127.12.201.195:49793 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 03 08:09:07 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.193@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
Sep 03 08:09:07 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.193@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
I20250903 08:09:07.678804 22999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:09:07.673552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:42385 (local address 127.12.201.193:45755)
0903 08:09:07.673847 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0903 08:09:07.673861 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0903 08:09:07.673953 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:09:07.674193 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:09:07.674195 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:09:07.674376 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:09:07.674573 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:09:07.674580 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:09:07.675431 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:09:07.675435 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:09:07.675990 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:09:07.676001 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:09:07.676136 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:07.676728 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:09:07.676743 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:09:07.678669 (+ 1926us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.201.254:42385: Server oryx/127.12.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":110,"threads_started":1}
W20250903 08:09:07.679054 23000 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 08:09:07.673641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.193:45755 (local address 127.12.201.254:42385)
0903 08:09:07.673913 (+ 272us) server_negotiation.cc:207] Beginning negotiation
0903 08:09:07.673917 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:09:07.673931 (+ 14us) server_negotiation.cc:408] Connection header received
0903 08:09:07.674037 (+ 106us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:09:07.674039 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:09:07.674084 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:09:07.674167 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:09:07.674675 (+ 508us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:07.675301 (+ 626us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:09:07.676115 (+ 814us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:07.676283 (+ 168us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:07.678834 (+ 2551us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.201.193:45755: BlockingRecv error: recv got EOF from 127.12.201.193:45755 (error 108)
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":67,"threads_started":1}
Sep 03 08:09:07 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.194@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
Sep 03 08:09:07 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.194@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
I20250903 08:09:07.852020 23001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:09:07.846779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:42385 (local address 127.12.201.194:40923)
0903 08:09:07.847143 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0903 08:09:07.847160 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0903 08:09:07.847289 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:09:07.847548 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:09:07.847551 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:09:07.847760 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:09:07.847957 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:09:07.847963 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:09:07.848697 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:09:07.848702 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:09:07.849075 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:09:07.849082 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:09:07.849199 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:07.849798 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:09:07.849812 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:09:07.851861 (+ 2049us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.201.254:42385: Server oryx/127.12.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":108,"threads_started":1}
W20250903 08:09:07.852186 23000 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 08:09:07.846902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.194:40923 (local address 127.12.201.254:42385)
0903 08:09:07.847121 (+ 219us) server_negotiation.cc:207] Beginning negotiation
0903 08:09:07.847126 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 08:09:07.847250 (+ 124us) server_negotiation.cc:408] Connection header received
0903 08:09:07.847392 (+ 142us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:09:07.847395 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:09:07.847445 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:09:07.847517 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:09:07.848087 (+ 570us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:07.848585 (+ 498us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:09:07.849193 (+ 608us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:07.849411 (+ 218us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:07.852000 (+ 2589us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.201.194:40923: BlockingRecv error: recv got EOF from 127.12.201.194:40923 (error 108)
Metrics: {"server-negotiator.queue_time_us":111}
Sep 03 08:09:08 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.195@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
Sep 03 08:09:08 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.195@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
I20250903 08:09:08.025477 23002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:09:08.020384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:42385 (local address 127.12.201.195:42685)
0903 08:09:08.020693 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0903 08:09:08.020708 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0903 08:09:08.020816 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:09:08.021082 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:09:08.021085 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:09:08.021296 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:09:08.021513 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:09:08.021519 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:09:08.022293 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:09:08.022297 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:09:08.022720 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:09:08.022727 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:09:08.022838 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:08.023520 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:09:08.023534 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:09:08.025351 (+ 1817us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.201.254:42385: Server oryx/127.12.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":99,"threads_started":1}
W20250903 08:09:08.025672 23000 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 08:09:08.020505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.195:42685 (local address 127.12.201.254:42385)
0903 08:09:08.020638 (+ 133us) server_negotiation.cc:207] Beginning negotiation
0903 08:09:08.020641 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 08:09:08.020826 (+ 185us) server_negotiation.cc:408] Connection header received
0903 08:09:08.020895 (+ 69us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:09:08.020897 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:09:08.020958 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:09:08.021033 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:09:08.021652 (+ 619us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:08.022156 (+ 504us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:09:08.022838 (+ 682us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:08.023004 (+ 166us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:08.025532 (+ 2528us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.201.195:42685: BlockingRecv error: recv got EOF from 127.12.201.195:42685 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 03 08:09:08 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.193@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
Sep 03 08:09:08 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.193@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
I20250903 08:09:08.689507 23003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:09:08.679722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:42385 (local address 127.12.201.193:38249)
0903 08:09:08.680014 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0903 08:09:08.680034 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0903 08:09:08.680136 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:09:08.680443 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:09:08.680446 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:09:08.680643 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:09:08.680868 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:09:08.680875 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:09:08.681772 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:09:08.681776 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:09:08.682320 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:09:08.682330 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:09:08.682447 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:08.687294 (+ 4847us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:09:08.687314 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:09:08.689351 (+ 2037us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.201.254:42385: Server oryx/127.12.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":91,"threads_started":1}
W20250903 08:09:08.689680 23004 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 08:09:08.679853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.193:38249 (local address 127.12.201.254:42385)
0903 08:09:08.680084 (+ 231us) server_negotiation.cc:207] Beginning negotiation
0903 08:09:08.680087 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 08:09:08.680103 (+ 16us) server_negotiation.cc:408] Connection header received
0903 08:09:08.680235 (+ 132us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:09:08.680238 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:09:08.680292 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:09:08.680375 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:09:08.681025 (+ 650us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:08.681598 (+ 573us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:09:08.682434 (+ 836us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:08.682646 (+ 212us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:08.689528 (+ 6882us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.201.193:38249: BlockingRecv error: recv got EOF from 127.12.201.193:38249 (error 108)
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":62,"threads_started":1}
Sep 03 08:09:08 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.194@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
Sep 03 08:09:08 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.194@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
I20250903 08:09:08.857693 23005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:09:08.852770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:42385 (local address 127.12.201.194:47331)
0903 08:09:08.853104 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0903 08:09:08.853117 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0903 08:09:08.853235 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:09:08.853485 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:09:08.853489 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:09:08.853695 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:09:08.853887 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:09:08.853893 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:09:08.854631 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:09:08.854635 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:09:08.855046 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:09:08.855053 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:09:08.855160 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:08.855792 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:09:08.855806 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:09:08.857573 (+ 1767us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.201.254:42385: Server oryx/127.12.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":91,"threads_started":1}
W20250903 08:09:08.857858 23004 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 08:09:08.852902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.194:47331 (local address 127.12.201.254:42385)
0903 08:09:08.853059 (+ 157us) server_negotiation.cc:207] Beginning negotiation
0903 08:09:08.853064 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 08:09:08.853200 (+ 136us) server_negotiation.cc:408] Connection header received
0903 08:09:08.853327 (+ 127us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:09:08.853333 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:09:08.853381 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:09:08.853450 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:09:08.854025 (+ 575us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:08.854520 (+ 495us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:09:08.855158 (+ 638us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:08.855384 (+ 226us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:08.857726 (+ 2342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.201.194:47331: BlockingRecv error: recv got EOF from 127.12.201.194:47331 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 03 08:09:09 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.195@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
Sep 03 08:09:09 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.195@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
I20250903 08:09:09.031126 23006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:09:09.026396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:42385 (local address 127.12.201.195:36827)
0903 08:09:09.026698 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0903 08:09:09.026715 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0903 08:09:09.026827 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:09:09.027133 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:09:09.027137 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:09:09.027342 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:09:09.027557 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:09:09.027563 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:09:09.028308 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:09:09.028311 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:09:09.028697 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:09:09.028704 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:09:09.028793 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:09.029345 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:09:09.029359 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:09:09.030974 (+ 1615us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.201.254:42385: Server oryx/127.12.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":95,"threads_started":1}
W20250903 08:09:09.031246 23004 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 08:09:09.026465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.195:36827 (local address 127.12.201.254:42385)
0903 08:09:09.026610 (+ 145us) server_negotiation.cc:207] Beginning negotiation
0903 08:09:09.026614 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:09:09.026814 (+ 200us) server_negotiation.cc:408] Connection header received
0903 08:09:09.026925 (+ 111us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:09:09.026928 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:09:09.027012 (+ 84us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:09:09.027109 (+ 97us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:09:09.027686 (+ 577us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:09.028186 (+ 500us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:09:09.028792 (+ 606us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:09.028924 (+ 132us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:09.031130 (+ 2206us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.201.195:36827: BlockingRecv error: recv got EOF from 127.12.201.195:36827 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 03 08:09:09 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.193@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
Sep 03 08:09:09 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.193@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
I20250903 08:09:09.695499 23007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:09:09.690366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:42385 (local address 127.12.201.193:59969)
0903 08:09:09.690704 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0903 08:09:09.690720 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0903 08:09:09.690830 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:09:09.691176 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:09:09.691181 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:09:09.691389 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:09:09.691617 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:09:09.691628 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:09:09.692535 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:09:09.692538 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:09:09.692924 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:09:09.692931 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:09:09.693031 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:09.693566 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:09:09.693578 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:09:09.695378 (+ 1800us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.201.254:42385: Server oryx/127.12.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":96,"threads_started":1}
W20250903 08:09:09.695636 23008 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 08:09:09.690503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.193:59969 (local address 127.12.201.254:42385)
0903 08:09:09.690731 (+ 228us) server_negotiation.cc:207] Beginning negotiation
0903 08:09:09.690735 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:09:09.690809 (+ 74us) server_negotiation.cc:408] Connection header received
0903 08:09:09.690930 (+ 121us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:09:09.690934 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:09:09.691013 (+ 79us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:09:09.691112 (+ 99us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:09:09.691741 (+ 629us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:09.692436 (+ 695us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:09:09.693033 (+ 597us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:09.693169 (+ 136us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:09.695514 (+ 2345us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.201.193:59969: BlockingRecv error: recv got EOF from 127.12.201.193:59969 (error 108)
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":61,"threads_started":1}
Sep 03 08:09:09 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.194@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
Sep 03 08:09:09 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.194@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
I20250903 08:09:09.863659 23009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:09:09.858527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:42385 (local address 127.12.201.194:49699)
0903 08:09:09.858836 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0903 08:09:09.858854 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0903 08:09:09.859020 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:09:09.859276 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:09:09.859279 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:09:09.859488 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:09:09.859682 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:09:09.859688 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:09:09.860417 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:09:09.860421 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:09:09.860793 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:09:09.860800 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:09:09.860907 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:09.861626 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:09:09.861641 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:09:09.863505 (+ 1864us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.201.254:42385: Server oryx/127.12.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":101,"threads_started":1}
W20250903 08:09:09.863770 23008 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 08:09:09.858709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.194:49699 (local address 127.12.201.254:42385)
0903 08:09:09.858858 (+ 149us) server_negotiation.cc:207] Beginning negotiation
0903 08:09:09.858863 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 08:09:09.858932 (+ 69us) server_negotiation.cc:408] Connection header received
0903 08:09:09.859127 (+ 195us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:09:09.859130 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:09:09.859176 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:09:09.859243 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:09:09.859807 (+ 564us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:09.860310 (+ 503us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:09:09.860906 (+ 596us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:09.861119 (+ 213us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:09.863660 (+ 2541us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.201.194:49699: BlockingRecv error: recv got EOF from 127.12.201.194:49699 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 03 08:09:10 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.195@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
Sep 03 08:09:10 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.195@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
I20250903 08:09:10.037432 23010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:09:10.032062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:42385 (local address 127.12.201.195:54867)
0903 08:09:10.032407 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0903 08:09:10.032424 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0903 08:09:10.032545 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:09:10.032846 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:09:10.032850 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:09:10.033105 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:09:10.033366 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:09:10.033378 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:09:10.034199 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:09:10.034202 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:09:10.034595 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:09:10.034601 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:09:10.034698 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:10.035346 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:09:10.035360 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:09:10.037311 (+ 1951us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.201.254:42385: Server oryx/127.12.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":82,"threads_started":1}
W20250903 08:09:10.037593 23008 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 08:09:10.032184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.195:54867 (local address 127.12.201.254:42385)
0903 08:09:10.032370 (+ 186us) server_negotiation.cc:207] Beginning negotiation
0903 08:09:10.032375 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 08:09:10.032498 (+ 123us) server_negotiation.cc:408] Connection header received
0903 08:09:10.032649 (+ 151us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:09:10.032655 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:09:10.032701 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:09:10.032782 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:09:10.033496 (+ 714us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:10.034097 (+ 601us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:09:10.034684 (+ 587us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:10.034807 (+ 123us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:10.037455 (+ 2648us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.201.195:54867: BlockingRecv error: recv got EOF from 127.12.201.195:54867 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 03 08:09:10 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.193@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
Sep 03 08:09:10 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.193@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
I20250903 08:09:10.701532 23011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:09:10.696416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:42385 (local address 127.12.201.193:51843)
0903 08:09:10.696714 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0903 08:09:10.696736 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0903 08:09:10.696835 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:09:10.697087 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:09:10.697089 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:09:10.697267 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:09:10.697471 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:09:10.697477 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:09:10.698316 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:09:10.698320 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:09:10.698859 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:09:10.698867 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:09:10.699002 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:10.699600 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:09:10.699613 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:09:10.701349 (+ 1736us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.201.254:42385: Server oryx/127.12.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":93,"threads_started":1}
W20250903 08:09:10.701602 23012 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 08:09:10.696536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.193:51843 (local address 127.12.201.254:42385)
0903 08:09:10.696812 (+ 276us) server_negotiation.cc:207] Beginning negotiation
0903 08:09:10.696815 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 08:09:10.696831 (+ 16us) server_negotiation.cc:408] Connection header received
0903 08:09:10.696922 (+ 91us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:09:10.696924 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:09:10.696971 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:09:10.697060 (+ 89us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:09:10.697607 (+ 547us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:10.698219 (+ 612us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:09:10.699014 (+ 795us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:10.699207 (+ 193us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:10.701484 (+ 2277us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.201.193:51843: BlockingRecv error: recv got EOF from 127.12.201.193:51843 (error 108)
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":68,"threads_started":1}
Sep 03 08:09:10 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.194@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
Sep 03 08:09:10 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.194@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
I20250903 08:09:10.869764 23013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:09:10.864524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:42385 (local address 127.12.201.194:47587)
0903 08:09:10.864807 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0903 08:09:10.864820 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0903 08:09:10.864916 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:09:10.865209 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:09:10.865212 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:09:10.865405 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:09:10.865602 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:09:10.865609 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:09:10.866464 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:09:10.866468 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:09:10.866943 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:09:10.866954 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:09:10.867091 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:10.867673 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:09:10.867685 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:09:10.869637 (+ 1952us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.201.254:42385: Server oryx/127.12.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":83,"threads_started":1}
W20250903 08:09:10.870004 23012 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 08:09:10.864651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.194:47587 (local address 127.12.201.254:42385)
0903 08:09:10.864789 (+ 138us) server_negotiation.cc:207] Beginning negotiation
0903 08:09:10.864793 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:09:10.864917 (+ 124us) server_negotiation.cc:408] Connection header received
0903 08:09:10.864998 (+ 81us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:09:10.865001 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:09:10.865053 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:09:10.865135 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:09:10.865741 (+ 606us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:10.866338 (+ 597us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:09:10.867133 (+ 795us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:10.867280 (+ 147us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:10.869878 (+ 2598us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.201.194:47587: BlockingRecv error: recv got EOF from 127.12.201.194:47587 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 03 08:09:11 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.195@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
Sep 03 08:09:11 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.195@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
I20250903 08:09:11.043422 23014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:09:11.038183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:42385 (local address 127.12.201.195:56231)
0903 08:09:11.038502 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0903 08:09:11.038517 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0903 08:09:11.038637 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:09:11.038878 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:09:11.038882 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:09:11.039135 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:09:11.039333 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:09:11.039340 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:09:11.040288 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:09:11.040292 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:09:11.040713 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:09:11.040721 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:09:11.040814 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:11.041359 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:09:11.041372 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:09:11.043244 (+ 1872us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.201.254:42385: Server oryx/127.12.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":113,"threads_started":1}
W20250903 08:09:11.043504 23012 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 08:09:11.038296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.195:56231 (local address 127.12.201.254:42385)
0903 08:09:11.038453 (+ 157us) server_negotiation.cc:207] Beginning negotiation
0903 08:09:11.038459 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0903 08:09:11.038596 (+ 137us) server_negotiation.cc:408] Connection header received
0903 08:09:11.038720 (+ 124us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:09:11.038723 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:09:11.038769 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:09:11.038845 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:09:11.039448 (+ 603us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:11.040176 (+ 728us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:09:11.040835 (+ 659us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:11.040982 (+ 147us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:11.043374 (+ 2392us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.201.195:56231: BlockingRecv error: recv got EOF from 127.12.201.195:56231 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 03 08:09:11 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.193@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
Sep 03 08:09:11 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.193@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
I20250903 08:09:11.707465 23015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:09:11.702454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:42385 (local address 127.12.201.193:54355)
0903 08:09:11.702746 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0903 08:09:11.702760 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0903 08:09:11.702853 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:09:11.703209 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:09:11.703211 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:09:11.703399 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:09:11.703596 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:09:11.703602 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:09:11.704371 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:09:11.704374 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:09:11.704823 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:09:11.704834 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:09:11.704953 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:11.705553 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:09:11.705570 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:09:11.707348 (+ 1778us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.201.254:42385: Server oryx/127.12.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":88,"threads_started":1}
W20250903 08:09:11.707751 23016 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 08:09:11.702559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.193:54355 (local address 127.12.201.254:42385)
0903 08:09:11.702799 (+ 240us) server_negotiation.cc:207] Beginning negotiation
0903 08:09:11.702803 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:09:11.702818 (+ 15us) server_negotiation.cc:408] Connection header received
0903 08:09:11.703017 (+ 199us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:09:11.703024 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:09:11.703083 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:09:11.703194 (+ 111us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:09:11.703718 (+ 524us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:11.704249 (+ 531us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:09:11.704942 (+ 693us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:11.705147 (+ 205us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:11.707611 (+ 2464us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.201.193:54355: BlockingRecv error: recv got EOF from 127.12.201.193:54355 (error 108)
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":60,"threads_started":1}
Sep 03 08:09:11 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.194@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
Sep 03 08:09:11 dist-test-slave-27r9 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.201.194@KRBTEST.COM for oryx/127.12.201.254@KRBTEST.COM, Server not found in Kerberos database
I20250903 08:09:11.875537 23017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:09:11.870688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:42385 (local address 127.12.201.194:49341)
0903 08:09:11.871019 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0903 08:09:11.871037 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0903 08:09:11.871160 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:09:11.871388 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:09:11.871390 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:09:11.871577 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:09:11.871769 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:09:11.871775 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:09:11.872503 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:09:11.872506 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:09:11.872873 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:09:11.872879 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:09:11.872964 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:11.873530 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:09:11.873544 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:09:11.875355 (+ 1811us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.201.254:42385: Server oryx/127.12.201.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":115,"threads_started":1}
W20250903 08:09:11.875597 23016 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 08:09:11.870758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.194:49341 (local address 127.12.201.254:42385)
0903 08:09:11.870917 (+ 159us) server_negotiation.cc:207] Beginning negotiation
0903 08:09:11.870922 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 08:09:11.871177 (+ 255us) server_negotiation.cc:408] Connection header received
0903 08:09:11.871240 (+ 63us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:09:11.871242 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:09:11.871292 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:09:11.871351 (+ 59us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:09:11.871903 (+ 552us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:11.872411 (+ 508us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:09:11.872989 (+ 578us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:11.873144 (+ 155us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:11.875490 (+ 2346us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.201.194:49341: BlockingRecv error: recv got EOF from 127.12.201.194:49341 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20250903 08:09:11.951623 13095 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUML90r/build/debug/bin/kudu with pid 22541
I20250903 08:09:11.957149 13095 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUML90r/build/debug/bin/kudu with pid 22693
I20250903 08:09:11.962280 13095 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUML90r/build/debug/bin/kudu with pid 22825
I20250903 08:09:11.967650 13095 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUML90r/build/debug/bin/kudu with pid 22471
2025-09-03T08:09:11Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18908 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2025-09-03T08:09:11Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-03T08:09:11Z Disabled control of system clock
I20250903 08:09:11.997074 13095 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUML90r/build/debug/bin/kudu
/tmp/dist-test-taskUML90r/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756886860253814-13095-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756886860253814-13095-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756886860253814-13095-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756886860253814-13095-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.201.254:43979
--webserver_interface=127.12.201.254
--webserver_port=0
--builtin_ntp_servers=127.12.201.212:41693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.201.254:43979
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20250903 08:09:12.104746 23021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:09:12.104981 23021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:09:12.105024 23021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:09:12.108322 23021 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250903 08:09:12.108392 23021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 08:09:12.108409 23021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:09:12.108429 23021 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250903 08:09:12.108446 23021 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250903 08:09:12.112274 23021 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.201.212:41693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756886860253814-13095-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756886860253814-13095-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.201.254:43979
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.201.254:43979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756886860253814-13095-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.201.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23021
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756886860253814-13095-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 07:50:08 UTC on 5fd53c4cbb9d
build id 7874
I20250903 08:09:12.113350 23021 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:09:12.114457 23021 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:09:12.120373 23026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:09:12.120379 23027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:09:12.120628 23029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:09:12.120865 23021 server_base.cc:1047] running on GCE node
I20250903 08:09:12.121507 23021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:09:12.122438 23021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:09:12.123600 23021 hybrid_clock.cc:648] HybridClock initialized: now 1756886952123576 us; error 39 us; skew 500 ppm
I20250903 08:09:12.125522 23021 webserver.cc:480] Webserver started at http://127.12.201.254:45091/ using document root <none> and password file <none>
I20250903 08:09:12.126112 23021 fs_manager.cc:362] Metadata directory not provided
I20250903 08:09:12.126175 23021 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:09:12.126363 23021 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:09:12.128314 23021 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756886860253814-13095-0/minicluster-data/master-0/data/instance:
uuid: "a0315d47dfba4728aa9a847249a578a3"
format_stamp: "Formatted at 2025-09-03 08:09:12 on dist-test-slave-27r9"
server_key: "66506097631124d54d8f0e8c5a7aa459"
server_key_iv: "f7667f4cafbc7b746c89669945c49f47"
server_key_version: "encryptionkey@0"
I20250903 08:09:12.128818 23021 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756886860253814-13095-0/minicluster-data/master-0/wal/instance:
uuid: "a0315d47dfba4728aa9a847249a578a3"
format_stamp: "Formatted at 2025-09-03 08:09:12 on dist-test-slave-27r9"
server_key: "66506097631124d54d8f0e8c5a7aa459"
server_key_iv: "f7667f4cafbc7b746c89669945c49f47"
server_key_version: "encryptionkey@0"
I20250903 08:09:12.132397 23021 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250903 08:09:12.134600 23035 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:09:12.135624 23021 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250903 08:09:12.135735 23021 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756886860253814-13095-0/minicluster-data/master-0/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756886860253814-13095-0/minicluster-data/master-0/wal
uuid: "a0315d47dfba4728aa9a847249a578a3"
format_stamp: "Formatted at 2025-09-03 08:09:12 on dist-test-slave-27r9"
server_key: "66506097631124d54d8f0e8c5a7aa459"
server_key_iv: "f7667f4cafbc7b746c89669945c49f47"
server_key_version: "encryptionkey@0"
I20250903 08:09:12.135824 23021 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756886860253814-13095-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756886860253814-13095-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756886860253814-13095-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:09:12.155270 23021 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:09:12.155984 23021 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:09:12.156184 23021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:09:12.163666 23021 rpc_server.cc:307] RPC server started. Bound to: 127.12.201.254:43979
I20250903 08:09:12.163671 23087 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.201.254:43979 every 8 connection(s)
I20250903 08:09:12.164762 23021 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756886860253814-13095-0/minicluster-data/master-0/data/info.pb
I20250903 08:09:12.167551 23088 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:09:12.172459 13095 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUML90r/build/debug/bin/kudu as pid 23021
I20250903 08:09:12.172590 13095 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756886860253814-13095-0/minicluster-data/master-0/wal/instance
I20250903 08:09:12.172857 13095 external_mini_cluster.cc:1468] Setting key 4c7a4abd493b0eff67a524a670508e73
I20250903 08:09:12.173655 23088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0315d47dfba4728aa9a847249a578a3: Bootstrap starting.
I20250903 08:09:12.176203 23088 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a0315d47dfba4728aa9a847249a578a3: Neither blocks nor log segments found. Creating new log.
I20250903 08:09:12.177155 23088 log.cc:826] T 00000000000000000000000000000000 P a0315d47dfba4728aa9a847249a578a3: Log is configured to *not* fsync() on all Append() calls
I20250903 08:09:12.179419 23088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0315d47dfba4728aa9a847249a578a3: No bootstrap required, opened a new log
I20250903 08:09:12.180603 23091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:09:12.174122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52374 (local address 127.12.201.254:43979)
0903 08:09:12.174746 (+ 624us) server_negotiation.cc:207] Beginning negotiation
0903 08:09:12.174755 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0903 08:09:12.174792 (+ 37us) server_negotiation.cc:408] Connection header received
0903 08:09:12.175351 (+ 559us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:09:12.175369 (+ 18us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:09:12.175740 (+ 371us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:09:12.176022 (+ 282us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:09:12.176609 (+ 587us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:12.177411 (+ 802us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:09:12.178157 (+ 746us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:12.178370 (+ 213us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:12.179023 (+ 653us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:09:12.179046 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:09:12.179058 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0903 08:09:12.179453 (+ 395us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:09:12.179494 (+ 41us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0903 08:09:12.179509 (+ 15us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:09:12.179620 (+ 111us) server_negotiation.cc:1036] Waiting for connection context
0903 08:09:12.179776 (+ 156us) server_negotiation.cc:300] Negotiation successful
0903 08:09:12.180016 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":127,"threads_started":1}
I20250903 08:09:12.182272 23088 raft_consensus.cc:357] T 00000000000000000000000000000000 P a0315d47dfba4728aa9a847249a578a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0315d47dfba4728aa9a847249a578a3" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 43979 } }
I20250903 08:09:12.182536 23088 raft_consensus.cc:383] T 00000000000000000000000000000000 P a0315d47dfba4728aa9a847249a578a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:09:12.182637 23088 raft_consensus.cc:738] T 00000000000000000000000000000000 P a0315d47dfba4728aa9a847249a578a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0315d47dfba4728aa9a847249a578a3, State: Initialized, Role: FOLLOWER
I20250903 08:09:12.183111 23088 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0315d47dfba4728aa9a847249a578a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0315d47dfba4728aa9a847249a578a3" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 43979 } }
I20250903 08:09:12.183231 23088 raft_consensus.cc:397] T 00000000000000000000000000000000 P a0315d47dfba4728aa9a847249a578a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250903 08:09:12.183279 23088 raft_consensus.cc:491] T 00000000000000000000000000000000 P a0315d47dfba4728aa9a847249a578a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250903 08:09:12.183359 23088 raft_consensus.cc:3058] T 00000000000000000000000000000000 P a0315d47dfba4728aa9a847249a578a3 [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:09:12.184198 23088 raft_consensus.cc:513] T 00000000000000000000000000000000 P a0315d47dfba4728aa9a847249a578a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0315d47dfba4728aa9a847249a578a3" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 43979 } }
I20250903 08:09:12.184494 23088 leader_election.cc:304] T 00000000000000000000000000000000 P a0315d47dfba4728aa9a847249a578a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a0315d47dfba4728aa9a847249a578a3; no voters:
I20250903 08:09:12.184744 23088 leader_election.cc:290] T 00000000000000000000000000000000 P a0315d47dfba4728aa9a847249a578a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250903 08:09:12.184870 23093 raft_consensus.cc:2802] T 00000000000000000000000000000000 P a0315d47dfba4728aa9a847249a578a3 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 08:09:12.185067 23093 raft_consensus.cc:695] T 00000000000000000000000000000000 P a0315d47dfba4728aa9a847249a578a3 [term 1 LEADER]: Becoming Leader. State: Replica: a0315d47dfba4728aa9a847249a578a3, State: Running, Role: LEADER
I20250903 08:09:12.185509 23093 consensus_queue.cc:237] T 00000000000000000000000000000000 P a0315d47dfba4728aa9a847249a578a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0315d47dfba4728aa9a847249a578a3" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 43979 } }
I20250903 08:09:12.185910 23088 sys_catalog.cc:564] T 00000000000000000000000000000000 P a0315d47dfba4728aa9a847249a578a3 [sys.catalog]: configured and running, proceeding with master startup.
I20250903 08:09:12.187302 23094 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0315d47dfba4728aa9a847249a578a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a0315d47dfba4728aa9a847249a578a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0315d47dfba4728aa9a847249a578a3" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 43979 } } }
I20250903 08:09:12.187428 23094 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0315d47dfba4728aa9a847249a578a3 [sys.catalog]: This master's current role is: LEADER
I20250903 08:09:12.187834 23102 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250903 08:09:12.187793 23095 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0315d47dfba4728aa9a847249a578a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0315d47dfba4728aa9a847249a578a3. Latest consensus state: current_term: 1 leader_uuid: "a0315d47dfba4728aa9a847249a578a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0315d47dfba4728aa9a847249a578a3" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 43979 } } }
I20250903 08:09:12.187984 23095 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0315d47dfba4728aa9a847249a578a3 [sys.catalog]: This master's current role is: LEADER
I20250903 08:09:12.190634 23102 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250903 08:09:12.195988 23102 catalog_manager.cc:1349] Generated new cluster ID: b9eeeb65ac654dc2931d214df29e9b3c
I20250903 08:09:12.196060 23102 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250903 08:09:12.221999 23102 catalog_manager.cc:1372] Generated new certificate authority record
I20250903 08:09:12.222896 23102 catalog_manager.cc:1506] Loading token signing keys...
I20250903 08:09:12.236362 23102 catalog_manager.cc:5955] T 00000000000000000000000000000000 P a0315d47dfba4728aa9a847249a578a3: Generated new TSK 0
I20250903 08:09:12.237062 23102 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250903 08:09:12.241900 13095 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUML90r/build/debug/bin/kudu
/tmp/dist-test-taskUML90r/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.201.193:0
--local_ip_for_outbound_sockets=127.12.201.193
--webserver_interface=127.12.201.193
--webserver_port=0
--tserver_master_addrs=127.12.201.254:43979
--builtin_ntp_servers=127.12.201.212:41693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20250903 08:09:12.350968 23112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:09:12.351276 23112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:09:12.351326 23112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:09:12.354565 23112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 08:09:12.354624 23112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:09:12.354696 23112 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.201.193
I20250903 08:09:12.358577 23112 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.201.212:41693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.201.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.201.193
--webserver_port=0
--tserver_master_addrs=127.12.201.254:43979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23112
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.201.193
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 07:50:08 UTC on 5fd53c4cbb9d
build id 7874
I20250903 08:09:12.359805 23112 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:09:12.360898 23112 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:09:12.367470 23120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:09:12.367421 23117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:09:12.367544 23112 server_base.cc:1047] running on GCE node
W20250903 08:09:12.367429 23118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:09:12.368026 23112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:09:12.368583 23112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:09:12.369722 23112 hybrid_clock.cc:648] HybridClock initialized: now 1756886952369704 us; error 34 us; skew 500 ppm
I20250903 08:09:12.371552 23112 webserver.cc:480] Webserver started at http://127.12.201.193:42113/ using document root <none> and password file <none>
I20250903 08:09:12.372107 23112 fs_manager.cc:362] Metadata directory not provided
I20250903 08:09:12.372161 23112 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:09:12.372340 23112 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:09:12.374050 23112 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-0/data/instance:
uuid: "7b8f8c72ea194eea83b75a1a04dbe41c"
format_stamp: "Formatted at 2025-09-03 08:09:12 on dist-test-slave-27r9"
server_key: "d4c162d20b129bc7d60c869d04df2876"
server_key_iv: "44a67cb2bd82e502f9a654a6cb1413ae"
server_key_version: "encryptionkey@0"
I20250903 08:09:12.374522 23112 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-0/wal/instance:
uuid: "7b8f8c72ea194eea83b75a1a04dbe41c"
format_stamp: "Formatted at 2025-09-03 08:09:12 on dist-test-slave-27r9"
server_key: "d4c162d20b129bc7d60c869d04df2876"
server_key_iv: "44a67cb2bd82e502f9a654a6cb1413ae"
server_key_version: "encryptionkey@0"
I20250903 08:09:12.377931 23112 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250903 08:09:12.380316 23126 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:09:12.381340 23112 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250903 08:09:12.381459 23112 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-0/wal
uuid: "7b8f8c72ea194eea83b75a1a04dbe41c"
format_stamp: "Formatted at 2025-09-03 08:09:12 on dist-test-slave-27r9"
server_key: "d4c162d20b129bc7d60c869d04df2876"
server_key_iv: "44a67cb2bd82e502f9a654a6cb1413ae"
server_key_version: "encryptionkey@0"
I20250903 08:09:12.381559 23112 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:09:12.401281 23112 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:09:12.402052 23112 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:09:12.402288 23112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:09:12.402945 23112 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 08:09:12.404193 23112 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 08:09:12.404284 23112 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 08:09:12.404352 23112 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 08:09:12.404388 23112 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 08:09:12.415314 23112 rpc_server.cc:307] RPC server started. Bound to: 127.12.201.193:40251
I20250903 08:09:12.415342 23239 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.201.193:40251 every 8 connection(s)
I20250903 08:09:12.416378 23112 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-0/data/info.pb
I20250903 08:09:12.419473 13095 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUML90r/build/debug/bin/kudu as pid 23112
I20250903 08:09:12.419605 13095 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-0/wal/instance
I20250903 08:09:12.419855 13095 external_mini_cluster.cc:1468] Setting key feeb48f82138b1edfc26acb72ef5025c
I20250903 08:09:12.421665 13095 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUML90r/build/debug/bin/kudu
/tmp/dist-test-taskUML90r/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.201.194:0
--local_ip_for_outbound_sockets=127.12.201.194
--webserver_interface=127.12.201.194
--webserver_port=0
--tserver_master_addrs=127.12.201.254:43979
--builtin_ntp_servers=127.12.201.212:41693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20250903 08:09:12.424646 23091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:09:12.418165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.193:44377 (local address 127.12.201.254:43979)
0903 08:09:12.418323 (+ 158us) server_negotiation.cc:207] Beginning negotiation
0903 08:09:12.418326 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 08:09:12.419284 (+ 958us) server_negotiation.cc:408] Connection header received
0903 08:09:12.420235 (+ 951us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:09:12.420239 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:09:12.420302 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:09:12.420387 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:09:12.421385 (+ 998us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:12.421939 (+ 554us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:09:12.422586 (+ 647us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:12.422764 (+ 178us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:12.424120 (+ 1356us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:09:12.424139 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:09:12.424143 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0903 08:09:12.424159 (+ 16us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:09:12.424182 (+ 23us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0903 08:09:12.424191 (+ 9us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:09:12.424264 (+ 73us) server_negotiation.cc:1036] Waiting for connection context
0903 08:09:12.424427 (+ 163us) server_negotiation.cc:300] Negotiation successful
0903 08:09:12.424500 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20250903 08:09:12.425372 23242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:09:12.418426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:43979 (local address 127.12.201.193:44377)
0903 08:09:12.419121 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0903 08:09:12.419160 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0903 08:09:12.420028 (+ 868us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:09:12.420568 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:09:12.420576 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:09:12.420661 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:09:12.421248 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:09:12.421260 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:09:12.422066 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:09:12.422069 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:09:12.422475 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:09:12.422482 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:09:12.422681 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:12.423879 (+ 1198us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0903 08:09:12.423881 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:09:12.423915 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0903 08:09:12.423974 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:09:12.424276 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:09:12.424286 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:09:12.424294 (+ 8us) client_negotiation.cc:769] Sending connection context
0903 08:09:12.424427 (+ 133us) client_negotiation.cc:240] Negotiation successful
0903 08:09:12.424685 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":103,"threads_started":1}
I20250903 08:09:12.426687 23240 heartbeater.cc:344] Connected to a master server at 127.12.201.254:43979
I20250903 08:09:12.426939 23240 heartbeater.cc:461] Registering TS with master...
I20250903 08:09:12.427562 23240 heartbeater.cc:507] Master 127.12.201.254:43979 requested a full tablet report, sending...
I20250903 08:09:12.429219 23052 ts_manager.cc:194] Registered new tserver with Master: 7b8f8c72ea194eea83b75a1a04dbe41c (127.12.201.193:40251)
I20250903 08:09:12.431046 23052 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.201.193:44377
W20250903 08:09:12.531978 23243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:09:12.532207 23243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:09:12.532256 23243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:09:12.535542 23243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 08:09:12.535602 23243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:09:12.535676 23243 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.201.194
I20250903 08:09:12.539685 23243 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.201.212:41693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.201.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.201.194
--webserver_port=0
--tserver_master_addrs=127.12.201.254:43979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23243
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.201.194
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 07:50:08 UTC on 5fd53c4cbb9d
build id 7874
I20250903 08:09:12.540795 23243 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:09:12.541894 23243 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:09:12.548313 23251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:09:12.548313 23249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:09:12.548367 23248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:09:12.548629 23243 server_base.cc:1047] running on GCE node
I20250903 08:09:12.548985 23243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:09:12.549536 23243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:09:12.550690 23243 hybrid_clock.cc:648] HybridClock initialized: now 1756886952550677 us; error 28 us; skew 500 ppm
I20250903 08:09:12.552533 23243 webserver.cc:480] Webserver started at http://127.12.201.194:37763/ using document root <none> and password file <none>
I20250903 08:09:12.553083 23243 fs_manager.cc:362] Metadata directory not provided
I20250903 08:09:12.553139 23243 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:09:12.553339 23243 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:09:12.555116 23243 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-1/data/instance:
uuid: "635421a1de1b40a6b75c31b627dba73b"
format_stamp: "Formatted at 2025-09-03 08:09:12 on dist-test-slave-27r9"
server_key: "80ef4f96433c4243292afc5d7492f8a9"
server_key_iv: "4685a4343cfbef04f533eb4551c1e14e"
server_key_version: "encryptionkey@0"
I20250903 08:09:12.555598 23243 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-1/wal/instance:
uuid: "635421a1de1b40a6b75c31b627dba73b"
format_stamp: "Formatted at 2025-09-03 08:09:12 on dist-test-slave-27r9"
server_key: "80ef4f96433c4243292afc5d7492f8a9"
server_key_iv: "4685a4343cfbef04f533eb4551c1e14e"
server_key_version: "encryptionkey@0"
I20250903 08:09:12.558912 23243 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20250903 08:09:12.561265 23257 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:09:12.562268 23243 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250903 08:09:12.562359 23243 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-1/wal
uuid: "635421a1de1b40a6b75c31b627dba73b"
format_stamp: "Formatted at 2025-09-03 08:09:12 on dist-test-slave-27r9"
server_key: "80ef4f96433c4243292afc5d7492f8a9"
server_key_iv: "4685a4343cfbef04f533eb4551c1e14e"
server_key_version: "encryptionkey@0"
I20250903 08:09:12.562469 23243 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:09:12.573047 23243 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:09:12.573587 23243 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:09:12.573743 23243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:09:12.574252 23243 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 08:09:12.575176 23243 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 08:09:12.575224 23243 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 08:09:12.575271 23243 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 08:09:12.575301 23243 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 08:09:12.583611 23243 rpc_server.cc:307] RPC server started. Bound to: 127.12.201.194:40621
I20250903 08:09:12.583660 23370 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.201.194:40621 every 8 connection(s)
I20250903 08:09:12.584566 23243 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-1/data/info.pb
I20250903 08:09:12.587643 13095 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUML90r/build/debug/bin/kudu as pid 23243
I20250903 08:09:12.587765 13095 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-1/wal/instance
I20250903 08:09:12.588045 13095 external_mini_cluster.cc:1468] Setting key aac565bc691668690300d6775eb8d283
I20250903 08:09:12.589993 13095 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUML90r/build/debug/bin/kudu
/tmp/dist-test-taskUML90r/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.201.195:0
--local_ip_for_outbound_sockets=127.12.201.195
--webserver_interface=127.12.201.195
--webserver_port=0
--tserver_master_addrs=127.12.201.254:43979
--builtin_ntp_servers=127.12.201.212:41693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20250903 08:09:12.592967 23091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:09:12.586181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.194:41643 (local address 127.12.201.254:43979)
0903 08:09:12.586363 (+ 182us) server_negotiation.cc:207] Beginning negotiation
0903 08:09:12.586368 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 08:09:12.587318 (+ 950us) server_negotiation.cc:408] Connection header received
0903 08:09:12.588373 (+ 1055us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:09:12.588376 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:09:12.588436 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:09:12.588529 (+ 93us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:09:12.589588 (+ 1059us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:12.590090 (+ 502us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:09:12.590887 (+ 797us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:12.591149 (+ 262us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:12.592388 (+ 1239us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:09:12.592410 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:09:12.592417 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0903 08:09:12.592435 (+ 18us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:09:12.592461 (+ 26us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0903 08:09:12.592473 (+ 12us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:09:12.592560 (+ 87us) server_negotiation.cc:1036] Waiting for connection context
0903 08:09:12.592686 (+ 126us) server_negotiation.cc:300] Negotiation successful
0903 08:09:12.592754 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20250903 08:09:12.593570 23373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:09:12.586518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:43979 (local address 127.12.201.194:41643)
0903 08:09:12.587195 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0903 08:09:12.587231 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0903 08:09:12.588169 (+ 938us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:09:12.588660 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:09:12.588667 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:09:12.588752 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:09:12.589451 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:09:12.589469 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:09:12.590258 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:09:12.590261 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:09:12.590754 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:09:12.590766 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:09:12.591010 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:12.592152 (+ 1142us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0903 08:09:12.592154 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:09:12.592184 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0903 08:09:12.592245 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:09:12.592558 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:09:12.592562 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:09:12.592566 (+ 4us) client_negotiation.cc:769] Sending connection context
0903 08:09:12.592669 (+ 103us) client_negotiation.cc:240] Negotiation successful
0903 08:09:12.592851 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":105,"threads_started":1}
I20250903 08:09:12.594625 23371 heartbeater.cc:344] Connected to a master server at 127.12.201.254:43979
I20250903 08:09:12.594847 23371 heartbeater.cc:461] Registering TS with master...
I20250903 08:09:12.595423 23371 heartbeater.cc:507] Master 127.12.201.254:43979 requested a full tablet report, sending...
I20250903 08:09:12.596371 23052 ts_manager.cc:194] Registered new tserver with Master: 635421a1de1b40a6b75c31b627dba73b (127.12.201.194:40621)
I20250903 08:09:12.596926 23052 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.201.194:41643
W20250903 08:09:12.696964 23374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:09:12.697186 23374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:09:12.697234 23374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:09:12.700469 23374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 08:09:12.700525 23374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:09:12.700613 23374 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.201.195
I20250903 08:09:12.704373 23374 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.201.212:41693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.201.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.201.195
--webserver_port=0
--tserver_master_addrs=127.12.201.254:43979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23374
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.201.195
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 07:50:08 UTC on 5fd53c4cbb9d
build id 7874
I20250903 08:09:12.705387 23374 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:09:12.706501 23374 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:09:12.712908 23379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:09:12.712908 23382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:09:12.713055 23380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:09:12.713294 23374 server_base.cc:1047] running on GCE node
I20250903 08:09:12.713668 23374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:09:12.714156 23374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:09:12.715312 23374 hybrid_clock.cc:648] HybridClock initialized: now 1756886952715298 us; error 37 us; skew 500 ppm
I20250903 08:09:12.717053 23374 webserver.cc:480] Webserver started at http://127.12.201.195:39189/ using document root <none> and password file <none>
I20250903 08:09:12.717603 23374 fs_manager.cc:362] Metadata directory not provided
I20250903 08:09:12.717664 23374 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:09:12.717842 23374 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:09:12.719620 23374 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-2/data/instance:
uuid: "064066eacdb24df1824dea156a8e9108"
format_stamp: "Formatted at 2025-09-03 08:09:12 on dist-test-slave-27r9"
server_key: "2c8a819454394de22d73794934706ee4"
server_key_iv: "a530570dcc9e82ab612eee846a1c372c"
server_key_version: "encryptionkey@0"
I20250903 08:09:12.720083 23374 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-2/wal/instance:
uuid: "064066eacdb24df1824dea156a8e9108"
format_stamp: "Formatted at 2025-09-03 08:09:12 on dist-test-slave-27r9"
server_key: "2c8a819454394de22d73794934706ee4"
server_key_iv: "a530570dcc9e82ab612eee846a1c372c"
server_key_version: "encryptionkey@0"
I20250903 08:09:12.723439 23374 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20250903 08:09:12.725692 23388 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:09:12.726791 23374 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250903 08:09:12.726928 23374 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-2/wal
uuid: "064066eacdb24df1824dea156a8e9108"
format_stamp: "Formatted at 2025-09-03 08:09:12 on dist-test-slave-27r9"
server_key: "2c8a819454394de22d73794934706ee4"
server_key_iv: "a530570dcc9e82ab612eee846a1c372c"
server_key_version: "encryptionkey@0"
I20250903 08:09:12.727067 23374 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:09:12.747704 23374 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:09:12.748302 23374 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:09:12.748464 23374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:09:12.749022 23374 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 08:09:12.749970 23374 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 08:09:12.750020 23374 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 08:09:12.750088 23374 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 08:09:12.750114 23374 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 08:09:12.758720 23374 rpc_server.cc:307] RPC server started. Bound to: 127.12.201.195:39309
I20250903 08:09:12.758744 23501 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.201.195:39309 every 8 connection(s)
I20250903 08:09:12.759742 23374 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-2/data/info.pb
I20250903 08:09:12.765924 13095 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUML90r/build/debug/bin/kudu as pid 23374
I20250903 08:09:12.766047 13095 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-2/wal/instance
I20250903 08:09:12.766342 13095 external_mini_cluster.cc:1468] Setting key 06a0abbe7e1367c8075953631e5a44ce
I20250903 08:09:12.768175 23091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:09:12.761627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.195:53471 (local address 127.12.201.254:43979)
0903 08:09:12.761820 (+ 193us) server_negotiation.cc:207] Beginning negotiation
0903 08:09:12.761825 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 08:09:12.762685 (+ 860us) server_negotiation.cc:408] Connection header received
0903 08:09:12.763532 (+ 847us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:09:12.763535 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:09:12.763584 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:09:12.763663 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:09:12.764606 (+ 943us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:12.765104 (+ 498us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:09:12.765909 (+ 805us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:12.766099 (+ 190us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:12.767683 (+ 1584us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:09:12.767700 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:09:12.767704 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0903 08:09:12.767719 (+ 15us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:09:12.767740 (+ 21us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0903 08:09:12.767748 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:09:12.767811 (+ 63us) server_negotiation.cc:1036] Waiting for connection context
0903 08:09:12.767976 (+ 165us) server_negotiation.cc:300] Negotiation successful
0903 08:09:12.768037 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20250903 08:09:12.768870 23504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:09:12.761892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:43979 (local address 127.12.201.195:53471)
0903 08:09:12.762554 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0903 08:09:12.762589 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0903 08:09:12.763322 (+ 733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:09:12.763805 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:09:12.763813 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:09:12.763891 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:09:12.764467 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:09:12.764479 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:09:12.765285 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:09:12.765289 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:09:12.765752 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:09:12.765764 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:09:12.765984 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:12.767426 (+ 1442us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0903 08:09:12.767428 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:09:12.767467 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0903 08:09:12.767539 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:09:12.767825 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:09:12.767833 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:09:12.767839 (+ 6us) client_negotiation.cc:769] Sending connection context
0903 08:09:12.767977 (+ 138us) client_negotiation.cc:240] Negotiation successful
0903 08:09:12.768212 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":101,"threads_started":1}
I20250903 08:09:12.769913 23502 heartbeater.cc:344] Connected to a master server at 127.12.201.254:43979
I20250903 08:09:12.770157 23502 heartbeater.cc:461] Registering TS with master...
I20250903 08:09:12.770768 23502 heartbeater.cc:507] Master 127.12.201.254:43979 requested a full tablet report, sending...
I20250903 08:09:12.771744 23052 ts_manager.cc:194] Registered new tserver with Master: 064066eacdb24df1824dea156a8e9108 (127.12.201.195:39309)
I20250903 08:09:12.772285 23052 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.201.195:53471
I20250903 08:09:12.780303 13095 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20250903 08:09:12.782841 23091 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 08:09:12.782160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52378 (local address 127.12.201.254:43979)
0903 08:09:12.782290 (+ 130us) server_negotiation.cc:207] Beginning negotiation
0903 08:09:12.782293 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 08:09:12.782354 (+ 61us) server_negotiation.cc:408] Connection header received
0903 08:09:12.782456 (+ 102us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:09:12.782458 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:09:12.782502 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:09:12.782575 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:09:12.782740 (+ 165us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52378: BlockingRecv error: recv got EOF from 127.0.0.1:52378 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20250903 08:09:12.783118 13095 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUML90r/build/debug/bin/kudu with pid 23112
I20250903 08:09:12.788322 13095 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUML90r/build/debug/bin/kudu with pid 23243
I20250903 08:09:12.794090 13095 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUML90r/build/debug/bin/kudu with pid 23374
I20250903 08:09:12.799909 13095 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUML90r/build/debug/bin/kudu with pid 23021
2025-09-03T08:09:12Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (832 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2025-09-03T08:09:12Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-03T08:09:12Z Disabled control of system clock
I20250903 08:09:12.829798 13095 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUML90r/build/debug/bin/kudu
/tmp/dist-test-taskUML90r/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756886860253814-13095-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756886860253814-13095-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756886860253814-13095-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756886860253814-13095-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.201.254:43485
--webserver_interface=127.12.201.254
--webserver_port=0
--builtin_ntp_servers=127.12.201.212:39051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.201.254:43485
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20250903 08:09:12.936573 23513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:09:12.936841 23513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:09:12.936892 23513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:09:12.940356 23513 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250903 08:09:12.940436 23513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 08:09:12.940461 23513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:09:12.940481 23513 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250903 08:09:12.940500 23513 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250903 08:09:12.944598 23513 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.201.212:39051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756886860253814-13095-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756886860253814-13095-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.201.254:43485
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.201.254:43485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756886860253814-13095-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.201.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23513
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756886860253814-13095-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 07:50:08 UTC on 5fd53c4cbb9d
build id 7874
I20250903 08:09:12.945801 23513 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:09:12.947075 23513 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:09:12.952822 23518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:09:12.952822 23521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:09:12.952822 23519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:09:12.953154 23513 server_base.cc:1047] running on GCE node
I20250903 08:09:12.953681 23513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:09:12.954599 23513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:09:12.955775 23513 hybrid_clock.cc:648] HybridClock initialized: now 1756886952955759 us; error 31 us; skew 500 ppm
I20250903 08:09:12.957559 23513 webserver.cc:480] Webserver started at http://127.12.201.254:38885/ using document root <none> and password file <none>
I20250903 08:09:12.958107 23513 fs_manager.cc:362] Metadata directory not provided
I20250903 08:09:12.958168 23513 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:09:12.958354 23513 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:09:12.960122 23513 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756886860253814-13095-0/minicluster-data/master-0/data/instance:
uuid: "fac36536e2a94e0f8dc12fb04fe1ed74"
format_stamp: "Formatted at 2025-09-03 08:09:12 on dist-test-slave-27r9"
server_key: "66d92e7d297bf9e5ccf5ad1b1b931fb9"
server_key_iv: "b4b1ca9efc2b230a053dcfe796e70ea8"
server_key_version: "encryptionkey@0"
I20250903 08:09:12.960597 23513 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756886860253814-13095-0/minicluster-data/master-0/wal/instance:
uuid: "fac36536e2a94e0f8dc12fb04fe1ed74"
format_stamp: "Formatted at 2025-09-03 08:09:12 on dist-test-slave-27r9"
server_key: "66d92e7d297bf9e5ccf5ad1b1b931fb9"
server_key_iv: "b4b1ca9efc2b230a053dcfe796e70ea8"
server_key_version: "encryptionkey@0"
I20250903 08:09:12.964002 23513 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20250903 08:09:12.966356 23527 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:09:12.967427 23513 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250903 08:09:12.967547 23513 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756886860253814-13095-0/minicluster-data/master-0/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756886860253814-13095-0/minicluster-data/master-0/wal
uuid: "fac36536e2a94e0f8dc12fb04fe1ed74"
format_stamp: "Formatted at 2025-09-03 08:09:12 on dist-test-slave-27r9"
server_key: "66d92e7d297bf9e5ccf5ad1b1b931fb9"
server_key_iv: "b4b1ca9efc2b230a053dcfe796e70ea8"
server_key_version: "encryptionkey@0"
I20250903 08:09:12.967643 23513 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756886860253814-13095-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756886860253814-13095-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756886860253814-13095-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:09:12.972368 23513 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:09:12.972934 23513 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:09:12.973088 23513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:09:12.980144 23579 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.201.254:43485 every 8 connection(s)
I20250903 08:09:12.980141 23513 rpc_server.cc:307] RPC server started. Bound to: 127.12.201.254:43485
I20250903 08:09:12.981180 23513 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756886860253814-13095-0/minicluster-data/master-0/data/info.pb
I20250903 08:09:12.983713 23580 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:09:12.985057 13095 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUML90r/build/debug/bin/kudu as pid 23513
I20250903 08:09:12.985158 13095 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756886860253814-13095-0/minicluster-data/master-0/wal/instance
I20250903 08:09:12.985404 13095 external_mini_cluster.cc:1468] Setting key 4cf304570351d3cfe6df873131b93593
I20250903 08:09:12.990052 23580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fac36536e2a94e0f8dc12fb04fe1ed74: Bootstrap starting.
I20250903 08:09:12.990844 23583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:09:12.986682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50750 (local address 127.12.201.254:43485)
0903 08:09:12.987381 (+ 699us) server_negotiation.cc:207] Beginning negotiation
0903 08:09:12.987393 (+ 12us) server_negotiation.cc:400] Waiting for connection header
0903 08:09:12.987438 (+ 45us) server_negotiation.cc:408] Connection header received
0903 08:09:12.988118 (+ 680us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:09:12.988144 (+ 26us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:09:12.988558 (+ 414us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:09:12.988919 (+ 361us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:09:12.989621 (+ 702us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:09:12.989626 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:09:12.989638 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0903 08:09:12.989649 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:09:12.989696 (+ 47us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0903 08:09:12.989710 (+ 14us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:09:12.989850 (+ 140us) server_negotiation.cc:1036] Waiting for connection context
0903 08:09:12.989933 (+ 83us) server_negotiation.cc:300] Negotiation successful
0903 08:09:12.990159 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":114,"threads_started":1}
I20250903 08:09:12.993377 23580 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fac36536e2a94e0f8dc12fb04fe1ed74: Neither blocks nor log segments found. Creating new log.
I20250903 08:09:12.994040 23580 log.cc:826] T 00000000000000000000000000000000 P fac36536e2a94e0f8dc12fb04fe1ed74: Log is configured to *not* fsync() on all Append() calls
I20250903 08:09:12.995780 23580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fac36536e2a94e0f8dc12fb04fe1ed74: No bootstrap required, opened a new log
I20250903 08:09:12.998306 23580 raft_consensus.cc:357] T 00000000000000000000000000000000 P fac36536e2a94e0f8dc12fb04fe1ed74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fac36536e2a94e0f8dc12fb04fe1ed74" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 43485 } }
I20250903 08:09:12.998487 23580 raft_consensus.cc:383] T 00000000000000000000000000000000 P fac36536e2a94e0f8dc12fb04fe1ed74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:09:12.998533 23580 raft_consensus.cc:738] T 00000000000000000000000000000000 P fac36536e2a94e0f8dc12fb04fe1ed74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fac36536e2a94e0f8dc12fb04fe1ed74, State: Initialized, Role: FOLLOWER
I20250903 08:09:12.998878 23580 consensus_queue.cc:260] T 00000000000000000000000000000000 P fac36536e2a94e0f8dc12fb04fe1ed74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fac36536e2a94e0f8dc12fb04fe1ed74" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 43485 } }
I20250903 08:09:12.999001 23580 raft_consensus.cc:397] T 00000000000000000000000000000000 P fac36536e2a94e0f8dc12fb04fe1ed74 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250903 08:09:12.999063 23580 raft_consensus.cc:491] T 00000000000000000000000000000000 P fac36536e2a94e0f8dc12fb04fe1ed74 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250903 08:09:12.999145 23580 raft_consensus.cc:3058] T 00000000000000000000000000000000 P fac36536e2a94e0f8dc12fb04fe1ed74 [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:09:12.999945 23580 raft_consensus.cc:513] T 00000000000000000000000000000000 P fac36536e2a94e0f8dc12fb04fe1ed74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fac36536e2a94e0f8dc12fb04fe1ed74" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 43485 } }
I20250903 08:09:13.000283 23580 leader_election.cc:304] T 00000000000000000000000000000000 P fac36536e2a94e0f8dc12fb04fe1ed74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fac36536e2a94e0f8dc12fb04fe1ed74; no voters:
I20250903 08:09:13.000602 23580 leader_election.cc:290] T 00000000000000000000000000000000 P fac36536e2a94e0f8dc12fb04fe1ed74 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250903 08:09:13.000717 23585 raft_consensus.cc:2802] T 00000000000000000000000000000000 P fac36536e2a94e0f8dc12fb04fe1ed74 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 08:09:13.000921 23585 raft_consensus.cc:695] T 00000000000000000000000000000000 P fac36536e2a94e0f8dc12fb04fe1ed74 [term 1 LEADER]: Becoming Leader. State: Replica: fac36536e2a94e0f8dc12fb04fe1ed74, State: Running, Role: LEADER
I20250903 08:09:13.001271 23585 consensus_queue.cc:237] T 00000000000000000000000000000000 P fac36536e2a94e0f8dc12fb04fe1ed74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fac36536e2a94e0f8dc12fb04fe1ed74" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 43485 } }
I20250903 08:09:13.001708 23580 sys_catalog.cc:564] T 00000000000000000000000000000000 P fac36536e2a94e0f8dc12fb04fe1ed74 [sys.catalog]: configured and running, proceeding with master startup.
I20250903 08:09:13.002794 23587 sys_catalog.cc:455] T 00000000000000000000000000000000 P fac36536e2a94e0f8dc12fb04fe1ed74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fac36536e2a94e0f8dc12fb04fe1ed74. Latest consensus state: current_term: 1 leader_uuid: "fac36536e2a94e0f8dc12fb04fe1ed74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fac36536e2a94e0f8dc12fb04fe1ed74" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 43485 } } }
I20250903 08:09:13.002907 23587 sys_catalog.cc:458] T 00000000000000000000000000000000 P fac36536e2a94e0f8dc12fb04fe1ed74 [sys.catalog]: This master's current role is: LEADER
I20250903 08:09:13.003144 23586 sys_catalog.cc:455] T 00000000000000000000000000000000 P fac36536e2a94e0f8dc12fb04fe1ed74 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fac36536e2a94e0f8dc12fb04fe1ed74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fac36536e2a94e0f8dc12fb04fe1ed74" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 43485 } } }
I20250903 08:09:13.003286 23586 sys_catalog.cc:458] T 00000000000000000000000000000000 P fac36536e2a94e0f8dc12fb04fe1ed74 [sys.catalog]: This master's current role is: LEADER
I20250903 08:09:13.003484 23594 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250903 08:09:13.006932 23594 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250903 08:09:13.012082 23594 catalog_manager.cc:1349] Generated new cluster ID: 745cb6d2329c47eaa0e7754ca38bfbc1
I20250903 08:09:13.012173 23594 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250903 08:09:13.025143 23594 catalog_manager.cc:1372] Generated new certificate authority record
I20250903 08:09:13.025470 23594 catalog_manager.cc:1506] Loading token signing keys...
I20250903 08:09:13.033236 23594 catalog_manager.cc:5955] T 00000000000000000000000000000000 P fac36536e2a94e0f8dc12fb04fe1ed74: Generated new TSK 0
I20250903 08:09:13.033951 23594 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250903 08:09:13.042840 13095 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUML90r/build/debug/bin/kudu
/tmp/dist-test-taskUML90r/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.201.193:0
--local_ip_for_outbound_sockets=127.12.201.193
--webserver_interface=127.12.201.193
--webserver_port=0
--tserver_master_addrs=127.12.201.254:43485
--builtin_ntp_servers=127.12.201.212:39051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20250903 08:09:13.166903 23604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:09:13.167174 23604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:09:13.167256 23604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:09:13.170539 23604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 08:09:13.170599 23604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:09:13.170712 23604 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.201.193
I20250903 08:09:13.174592 23604 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.201.212:39051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.201.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.201.193
--webserver_port=0
--tserver_master_addrs=127.12.201.254:43485
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23604
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.201.193
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 07:50:08 UTC on 5fd53c4cbb9d
build id 7874
I20250903 08:09:13.175719 23604 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:09:13.176865 23604 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:09:13.183534 23609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:09:13.183516 23612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:09:13.183512 23610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:09:13.183951 23604 server_base.cc:1047] running on GCE node
I20250903 08:09:13.184356 23604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:09:13.185006 23604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:09:13.186188 23604 hybrid_clock.cc:648] HybridClock initialized: now 1756886953186166 us; error 36 us; skew 500 ppm
I20250903 08:09:13.187995 23604 webserver.cc:480] Webserver started at http://127.12.201.193:40885/ using document root <none> and password file <none>
I20250903 08:09:13.188557 23604 fs_manager.cc:362] Metadata directory not provided
I20250903 08:09:13.188614 23604 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:09:13.188810 23604 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:09:13.190518 23604 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-0/data/instance:
uuid: "2063c363256c42c0a4871751d3e2d1fc"
format_stamp: "Formatted at 2025-09-03 08:09:13 on dist-test-slave-27r9"
server_key: "404218f5cf2e683ef73fcb7f9ac20d37"
server_key_iv: "feb55355a8e8c33c8ff1a968c965ea7a"
server_key_version: "encryptionkey@0"
I20250903 08:09:13.190954 23604 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-0/wal/instance:
uuid: "2063c363256c42c0a4871751d3e2d1fc"
format_stamp: "Formatted at 2025-09-03 08:09:13 on dist-test-slave-27r9"
server_key: "404218f5cf2e683ef73fcb7f9ac20d37"
server_key_iv: "feb55355a8e8c33c8ff1a968c965ea7a"
server_key_version: "encryptionkey@0"
I20250903 08:09:13.194295 23604 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20250903 08:09:13.196550 23618 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:09:13.197628 23604 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250903 08:09:13.197731 23604 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-0/wal
uuid: "2063c363256c42c0a4871751d3e2d1fc"
format_stamp: "Formatted at 2025-09-03 08:09:13 on dist-test-slave-27r9"
server_key: "404218f5cf2e683ef73fcb7f9ac20d37"
server_key_iv: "feb55355a8e8c33c8ff1a968c965ea7a"
server_key_version: "encryptionkey@0"
I20250903 08:09:13.197814 23604 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:09:13.202679 23604 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:09:13.203385 23604 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:09:13.203549 23604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:09:13.204087 23604 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 08:09:13.205019 23604 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 08:09:13.205067 23604 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 08:09:13.205106 23604 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 08:09:13.205121 23604 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 08:09:13.213903 23604 rpc_server.cc:307] RPC server started. Bound to: 127.12.201.193:33379
I20250903 08:09:13.213948 23731 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.201.193:33379 every 8 connection(s)
I20250903 08:09:13.214897 23604 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-0/data/info.pb
I20250903 08:09:13.219357 13095 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUML90r/build/debug/bin/kudu as pid 23604
I20250903 08:09:13.219476 13095 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-0/wal/instance
I20250903 08:09:13.219776 13095 external_mini_cluster.cc:1468] Setting key 6a6832dfe5044214dd15e155b0e8271d
I20250903 08:09:13.220297 23583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:09:13.216718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.193:51777 (local address 127.12.201.254:43485)
0903 08:09:13.216866 (+ 148us) server_negotiation.cc:207] Beginning negotiation
0903 08:09:13.216870 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:09:13.217770 (+ 900us) server_negotiation.cc:408] Connection header received
0903 08:09:13.218562 (+ 792us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:09:13.218565 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:09:13.218607 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:09:13.218693 (+ 86us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:09:13.219833 (+ 1140us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:09:13.219837 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:09:13.219841 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0903 08:09:13.219858 (+ 17us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:09:13.219884 (+ 26us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0903 08:09:13.219893 (+ 9us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:09:13.220004 (+ 111us) server_negotiation.cc:1036] Waiting for connection context
0903 08:09:13.220132 (+ 128us) server_negotiation.cc:300] Negotiation successful
0903 08:09:13.220181 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20250903 08:09:13.221089 23734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:09:13.217001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:43485 (local address 127.12.201.193:51777)
0903 08:09:13.217634 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0903 08:09:13.217670 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0903 08:09:13.218368 (+ 698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:09:13.218832 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:09:13.218840 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:09:13.218926 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:09:13.219563 (+ 637us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0903 08:09:13.219565 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:09:13.219605 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0903 08:09:13.219682 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:09:13.219989 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:09:13.219995 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:09:13.220000 (+ 5us) client_negotiation.cc:769] Sending connection context
0903 08:09:13.220160 (+ 160us) client_negotiation.cc:240] Negotiation successful
0903 08:09:13.220416 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":113,"threads_started":1}
I20250903 08:09:13.221733 13095 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUML90r/build/debug/bin/kudu
/tmp/dist-test-taskUML90r/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.201.194:0
--local_ip_for_outbound_sockets=127.12.201.194
--webserver_interface=127.12.201.194
--webserver_port=0
--tserver_master_addrs=127.12.201.254:43485
--builtin_ntp_servers=127.12.201.212:39051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20250903 08:09:13.222244 23732 heartbeater.cc:344] Connected to a master server at 127.12.201.254:43485
I20250903 08:09:13.222512 23732 heartbeater.cc:461] Registering TS with master...
I20250903 08:09:13.223022 23732 heartbeater.cc:507] Master 127.12.201.254:43485 requested a full tablet report, sending...
I20250903 08:09:13.224610 23544 ts_manager.cc:194] Registered new tserver with Master: 2063c363256c42c0a4871751d3e2d1fc (127.12.201.193:33379)
W20250903 08:09:13.328384 23735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:09:13.328687 23735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:09:13.328783 23735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:09:13.332159 23735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 08:09:13.332229 23735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:09:13.332356 23735 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.201.194
I20250903 08:09:13.336202 23735 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.201.212:39051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.201.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.201.194
--webserver_port=0
--tserver_master_addrs=127.12.201.254:43485
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23735
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.201.194
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 07:50:08 UTC on 5fd53c4cbb9d
build id 7874
I20250903 08:09:13.337359 23735 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:09:13.338510 23735 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:09:13.344964 23743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:09:13.345098 23740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:09:13.345131 23741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:09:13.345196 23735 server_base.cc:1047] running on GCE node
I20250903 08:09:13.345664 23735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:09:13.346207 23735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:09:13.347373 23735 hybrid_clock.cc:648] HybridClock initialized: now 1756886953347350 us; error 44 us; skew 500 ppm
I20250903 08:09:13.349236 23735 webserver.cc:480] Webserver started at http://127.12.201.194:45605/ using document root <none> and password file <none>
I20250903 08:09:13.349824 23735 fs_manager.cc:362] Metadata directory not provided
I20250903 08:09:13.349885 23735 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:09:13.350070 23735 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:09:13.351863 23735 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-1/data/instance:
uuid: "835a903e58d24b268c222ca132dd3fde"
format_stamp: "Formatted at 2025-09-03 08:09:13 on dist-test-slave-27r9"
server_key: "325e14664f219f9166b54f84cb6be57c"
server_key_iv: "ba7216549729169d80c2224bbdc93d34"
server_key_version: "encryptionkey@0"
I20250903 08:09:13.352391 23735 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-1/wal/instance:
uuid: "835a903e58d24b268c222ca132dd3fde"
format_stamp: "Formatted at 2025-09-03 08:09:13 on dist-test-slave-27r9"
server_key: "325e14664f219f9166b54f84cb6be57c"
server_key_iv: "ba7216549729169d80c2224bbdc93d34"
server_key_version: "encryptionkey@0"
I20250903 08:09:13.355837 23735 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250903 08:09:13.358067 23749 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:09:13.359105 23735 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250903 08:09:13.359211 23735 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-1/wal
uuid: "835a903e58d24b268c222ca132dd3fde"
format_stamp: "Formatted at 2025-09-03 08:09:13 on dist-test-slave-27r9"
server_key: "325e14664f219f9166b54f84cb6be57c"
server_key_iv: "ba7216549729169d80c2224bbdc93d34"
server_key_version: "encryptionkey@0"
I20250903 08:09:13.359309 23735 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:09:13.363960 23735 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:09:13.364499 23735 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:09:13.364660 23735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:09:13.365161 23735 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 08:09:13.366046 23735 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 08:09:13.366091 23735 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 08:09:13.366132 23735 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 08:09:13.366165 23735 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 08:09:13.375432 23735 rpc_server.cc:307] RPC server started. Bound to: 127.12.201.194:39483
I20250903 08:09:13.375500 23862 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.201.194:39483 every 8 connection(s)
I20250903 08:09:13.376430 23735 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-1/data/info.pb
I20250903 08:09:13.377839 13095 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUML90r/build/debug/bin/kudu as pid 23735
I20250903 08:09:13.377941 13095 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-1/wal/instance
I20250903 08:09:13.378188 13095 external_mini_cluster.cc:1468] Setting key 18743e4c650bb5bb4c9f65aee141cf56
I20250903 08:09:13.380282 13095 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUML90r/build/debug/bin/kudu
/tmp/dist-test-taskUML90r/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.201.195:0
--local_ip_for_outbound_sockets=127.12.201.195
--webserver_interface=127.12.201.195
--webserver_port=0
--tserver_master_addrs=127.12.201.254:43485
--builtin_ntp_servers=127.12.201.212:39051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20250903 08:09:13.382536 23583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:09:13.378315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.194:34179 (local address 127.12.201.254:43485)
0903 08:09:13.378471 (+ 156us) server_negotiation.cc:207] Beginning negotiation
0903 08:09:13.378476 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 08:09:13.379506 (+ 1030us) server_negotiation.cc:408] Connection header received
0903 08:09:13.380668 (+ 1162us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:09:13.380671 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:09:13.380709 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:09:13.380791 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:09:13.382064 (+ 1273us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:09:13.382068 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:09:13.382070 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0903 08:09:13.382080 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:09:13.382099 (+ 19us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0903 08:09:13.382106 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:09:13.382185 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
0903 08:09:13.382334 (+ 149us) server_negotiation.cc:300] Negotiation successful
0903 08:09:13.382378 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20250903 08:09:13.383267 23865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:09:13.378647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:43485 (local address 127.12.201.194:34179)
0903 08:09:13.379381 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0903 08:09:13.379429 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0903 08:09:13.380344 (+ 915us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:09:13.380971 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:09:13.380985 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:09:13.381100 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:09:13.381831 (+ 731us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0903 08:09:13.381833 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:09:13.381867 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0903 08:09:13.381925 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:09:13.382178 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:09:13.382185 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:09:13.382192 (+ 7us) client_negotiation.cc:769] Sending connection context
0903 08:09:13.382376 (+ 184us) client_negotiation.cc:240] Negotiation successful
0903 08:09:13.382595 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":145,"threads_started":1}
I20250903 08:09:13.384321 23863 heartbeater.cc:344] Connected to a master server at 127.12.201.254:43485
I20250903 08:09:13.384554 23863 heartbeater.cc:461] Registering TS with master...
I20250903 08:09:13.384956 23863 heartbeater.cc:507] Master 127.12.201.254:43485 requested a full tablet report, sending...
I20250903 08:09:13.385936 23544 ts_manager.cc:194] Registered new tserver with Master: 835a903e58d24b268c222ca132dd3fde (127.12.201.194:39483)
W20250903 08:09:13.488555 23866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:09:13.488786 23866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:09:13.488839 23866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:09:13.492549 23866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 08:09:13.492616 23866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:09:13.492707 23866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.201.195
I20250903 08:09:13.496685 23866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.201.212:39051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.201.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.201.195
--webserver_port=0
--tserver_master_addrs=127.12.201.254:43485
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.201.195
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 07:50:08 UTC on 5fd53c4cbb9d
build id 7874
I20250903 08:09:13.497812 23866 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:09:13.498950 23866 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:09:13.505448 23871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:09:13.505399 23874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:09:13.505401 23872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:09:13.505656 23866 server_base.cc:1047] running on GCE node
I20250903 08:09:13.506019 23866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:09:13.506572 23866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:09:13.507718 23866 hybrid_clock.cc:648] HybridClock initialized: now 1756886953507706 us; error 29 us; skew 500 ppm
I20250903 08:09:13.509470 23866 webserver.cc:480] Webserver started at http://127.12.201.195:45965/ using document root <none> and password file <none>
I20250903 08:09:13.510030 23866 fs_manager.cc:362] Metadata directory not provided
I20250903 08:09:13.510090 23866 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:09:13.510265 23866 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:09:13.512022 23866 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-2/data/instance:
uuid: "842062e9fd534d03a7f6b7c63771212b"
format_stamp: "Formatted at 2025-09-03 08:09:13 on dist-test-slave-27r9"
server_key: "6a820598c0da60048be50af8d2acd46c"
server_key_iv: "99750f0c526c5a2dcb7f672072a67b03"
server_key_version: "encryptionkey@0"
I20250903 08:09:13.512495 23866 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-2/wal/instance:
uuid: "842062e9fd534d03a7f6b7c63771212b"
format_stamp: "Formatted at 2025-09-03 08:09:13 on dist-test-slave-27r9"
server_key: "6a820598c0da60048be50af8d2acd46c"
server_key_iv: "99750f0c526c5a2dcb7f672072a67b03"
server_key_version: "encryptionkey@0"
I20250903 08:09:13.515906 23866 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250903 08:09:13.518133 23880 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:09:13.519212 23866 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250903 08:09:13.519325 23866 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-2/wal
uuid: "842062e9fd534d03a7f6b7c63771212b"
format_stamp: "Formatted at 2025-09-03 08:09:13 on dist-test-slave-27r9"
server_key: "6a820598c0da60048be50af8d2acd46c"
server_key_iv: "99750f0c526c5a2dcb7f672072a67b03"
server_key_version: "encryptionkey@0"
I20250903 08:09:13.519418 23866 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:09:13.524142 23866 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:09:13.524691 23866 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:09:13.524844 23866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:09:13.525458 23866 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 08:09:13.526597 23866 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 08:09:13.526660 23866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 08:09:13.526731 23866 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 08:09:13.526763 23866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 08:09:13.535874 23866 rpc_server.cc:307] RPC server started. Bound to: 127.12.201.195:38451
I20250903 08:09:13.535905 23993 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.201.195:38451 every 8 connection(s)
I20250903 08:09:13.536919 23866 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-2/data/info.pb
I20250903 08:09:13.542676 23583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:09:13.538694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.195:40395 (local address 127.12.201.254:43485)
0903 08:09:13.538868 (+ 174us) server_negotiation.cc:207] Beginning negotiation
0903 08:09:13.538873 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 08:09:13.539967 (+ 1094us) server_negotiation.cc:408] Connection header received
0903 08:09:13.540792 (+ 825us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:09:13.540795 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:09:13.540837 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:09:13.540932 (+ 95us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:09:13.542126 (+ 1194us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:09:13.542129 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:09:13.542132 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0903 08:09:13.542145 (+ 13us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:09:13.542167 (+ 22us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0903 08:09:13.542178 (+ 11us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:09:13.542331 (+ 153us) server_negotiation.cc:1036] Waiting for connection context
0903 08:09:13.542478 (+ 147us) server_negotiation.cc:300] Negotiation successful
0903 08:09:13.542528 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20250903 08:09:13.543462 23996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:09:13.539126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:43485 (local address 127.12.201.195:40395)
0903 08:09:13.539818 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0903 08:09:13.539855 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0903 08:09:13.540595 (+ 740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:09:13.541133 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:09:13.541142 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:09:13.541228 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:09:13.541845 (+ 617us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0903 08:09:13.541847 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:09:13.541879 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0903 08:09:13.541935 (+ 56us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:09:13.542318 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:09:13.542330 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:09:13.542336 (+ 6us) client_negotiation.cc:769] Sending connection context
0903 08:09:13.542535 (+ 199us) client_negotiation.cc:240] Negotiation successful
0903 08:09:13.542757 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":124,"threads_started":1}
I20250903 08:09:13.544567 23994 heartbeater.cc:344] Connected to a master server at 127.12.201.254:43485
I20250903 08:09:13.544798 23994 heartbeater.cc:461] Registering TS with master...
I20250903 08:09:13.545181 23994 heartbeater.cc:507] Master 127.12.201.254:43485 requested a full tablet report, sending...
I20250903 08:09:13.546142 23544 ts_manager.cc:194] Registered new tserver with Master: 842062e9fd534d03a7f6b7c63771212b (127.12.201.195:38451)
I20250903 08:09:13.547107 13095 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUML90r/build/debug/bin/kudu as pid 23866
I20250903 08:09:13.547223 13095 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756886860253814-13095-0/minicluster-data/ts-2/wal/instance
I20250903 08:09:13.547462 13095 external_mini_cluster.cc:1468] Setting key 40a82fb2eaf04a2ea1cf20d2f886fe46
I20250903 08:09:13.549261 13095 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20250903 08:09:13.551684 23583 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 08:09:13.550949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50754 (local address 127.12.201.254:43485)
0903 08:09:13.551109 (+ 160us) server_negotiation.cc:207] Beginning negotiation
0903 08:09:13.551113 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:09:13.551207 (+ 94us) server_negotiation.cc:408] Connection header received
0903 08:09:13.551316 (+ 109us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:09:13.551319 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:09:13.551350 (+ 31us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:09:13.551414 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:09:13.551571 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50754: BlockingRecv error: recv got EOF from 127.0.0.1:50754 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20250903 08:09:13.551842 13095 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUML90r/build/debug/bin/kudu with pid 23604
I20250903 08:09:13.557564 13095 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUML90r/build/debug/bin/kudu with pid 23735
I20250903 08:09:13.562736 13095 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUML90r/build/debug/bin/kudu with pid 23866
I20250903 08:09:13.568151 13095 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUML90r/build/debug/bin/kudu with pid 23513
2025-09-03T08:09:13Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (766 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 03 08:09:13 dist-test-slave-27r9 krb5kdc[24004](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 03 08:09:13 dist-test-slave-27r9 krb5kdc[24004](info): set up 2 sockets
Sep 03 08:09:13 dist-test-slave-27r9 krb5kdc[24004](info): commencing operation
krb5kdc: starting...
W20250903 08:09:15.567796 13095 mini_kdc.cc:121] Time spent starting KDC: real 1.978s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/krb5kdc/test-user.keytab.
Sep 03 08:09:15 dist-test-slave-27r9 krb5kdc[24004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-09-03T08:09:15Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-03T08:09:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.201.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.201.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.201.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.201.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.201.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.201.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.201.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.201.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/krb5kdc/kudu.keytab.
I20250903 08:09:15.706781 13095 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUML90r/build/debug/bin/kudu
/tmp/dist-test-taskUML90r/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.201.254:42775
--webserver_interface=127.12.201.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.201.212:35551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.201.254:42775
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 08:09:15.815804 24020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:09:15.816049 24020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:09:15.816129 24020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:09:15.819350 24020 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250903 08:09:15.819429 24020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 08:09:15.819465 24020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:09:15.819509 24020 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250903 08:09:15.819541 24020 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250903 08:09:15.823921 24020 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.201.212:35551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.201.254:42775
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.201.254:42775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.201.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24020
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 07:50:08 UTC on 5fd53c4cbb9d
build id 7874
I20250903 08:09:15.825047 24020 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:09:15.825898 24020 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:09:15.831440 24028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:09:15.831494 24025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:09:15.831768 24020 server_base.cc:1047] running on GCE node
W20250903 08:09:15.831446 24026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:09:15.832273 24020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:09:15.833186 24020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:09:15.834363 24020 hybrid_clock.cc:648] HybridClock initialized: now 1756886955834348 us; error 29 us; skew 500 ppm
Sep 03 08:09:15 dist-test-slave-27r9 krb5kdc[24004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886955, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 08:09:15.837234 24020 init.cc:377] Logged in from keytab as kudu/127.12.201.254@KRBTEST.COM (short username kudu)
I20250903 08:09:15.838286 24020 webserver.cc:480] Webserver started at http://127.12.201.254:38297/ using document root <none> and password file <none>
I20250903 08:09:15.838830 24020 fs_manager.cc:362] Metadata directory not provided
I20250903 08:09:15.838877 24020 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:09:15.839079 24020 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:09:15.840857 24020 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/minicluster-data/master-0/data/instance:
uuid: "e2f3b5eb3fc34530b7e91cf02d665309"
format_stamp: "Formatted at 2025-09-03 08:09:15 on dist-test-slave-27r9"
server_key: "2684ea5b60b6d27bf08e0eeeaa5f3715"
server_key_iv: "a6948d20075869498c75bd63a3709f1a"
server_key_version: "encryptionkey@0"
I20250903 08:09:15.841301 24020 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/minicluster-data/master-0/wal/instance:
uuid: "e2f3b5eb3fc34530b7e91cf02d665309"
format_stamp: "Formatted at 2025-09-03 08:09:15 on dist-test-slave-27r9"
server_key: "2684ea5b60b6d27bf08e0eeeaa5f3715"
server_key_iv: "a6948d20075869498c75bd63a3709f1a"
server_key_version: "encryptionkey@0"
I20250903 08:09:15.844748 24020 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250903 08:09:15.847059 24035 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:09:15.848109 24020 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250903 08:09:15.848215 24020 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/minicluster-data/master-0/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/minicluster-data/master-0/wal
uuid: "e2f3b5eb3fc34530b7e91cf02d665309"
format_stamp: "Formatted at 2025-09-03 08:09:15 on dist-test-slave-27r9"
server_key: "2684ea5b60b6d27bf08e0eeeaa5f3715"
server_key_iv: "a6948d20075869498c75bd63a3709f1a"
server_key_version: "encryptionkey@0"
I20250903 08:09:15.848296 24020 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:09:15.886179 24020 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:09:15.894284 24020 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:09:15.894459 24020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:09:15.901890 24020 rpc_server.cc:307] RPC server started. Bound to: 127.12.201.254:42775
I20250903 08:09:15.901895 24087 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.201.254:42775 every 8 connection(s)
I20250903 08:09:15.902961 24020 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/minicluster-data/master-0/data/info.pb
I20250903 08:09:15.905652 24088 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:09:15.911602 24088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2f3b5eb3fc34530b7e91cf02d665309: Bootstrap starting.
I20250903 08:09:15.912985 13095 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUML90r/build/debug/bin/kudu as pid 24020
I20250903 08:09:15.913111 13095 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/minicluster-data/master-0/wal/instance
I20250903 08:09:15.913363 13095 external_mini_cluster.cc:1468] Setting key 0caec0714a9cf851daa424c480751d3f
I20250903 08:09:15.914461 24088 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e2f3b5eb3fc34530b7e91cf02d665309: Neither blocks nor log segments found. Creating new log.
I20250903 08:09:15.915273 24088 log.cc:826] T 00000000000000000000000000000000 P e2f3b5eb3fc34530b7e91cf02d665309: Log is configured to *not* fsync() on all Append() calls
I20250903 08:09:15.917601 24088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2f3b5eb3fc34530b7e91cf02d665309: No bootstrap required, opened a new log
I20250903 08:09:15.920914 24088 raft_consensus.cc:357] T 00000000000000000000000000000000 P e2f3b5eb3fc34530b7e91cf02d665309 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2f3b5eb3fc34530b7e91cf02d665309" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 42775 } }
Sep 03 08:09:15 dist-test-slave-27r9 krb5kdc[24004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.201.254@KRBTEST.COM
I20250903 08:09:15.921116 24088 raft_consensus.cc:383] T 00000000000000000000000000000000 P e2f3b5eb3fc34530b7e91cf02d665309 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:09:15.921188 24088 raft_consensus.cc:738] T 00000000000000000000000000000000 P e2f3b5eb3fc34530b7e91cf02d665309 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2f3b5eb3fc34530b7e91cf02d665309, State: Initialized, Role: FOLLOWER
I20250903 08:09:15.921631 24088 consensus_queue.cc:260] T 00000000000000000000000000000000 P e2f3b5eb3fc34530b7e91cf02d665309 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2f3b5eb3fc34530b7e91cf02d665309" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 42775 } }
I20250903 08:09:15.921761 24088 raft_consensus.cc:397] T 00000000000000000000000000000000 P e2f3b5eb3fc34530b7e91cf02d665309 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250903 08:09:15.921825 24088 raft_consensus.cc:491] T 00000000000000000000000000000000 P e2f3b5eb3fc34530b7e91cf02d665309 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250903 08:09:15.921940 24088 raft_consensus.cc:3058] T 00000000000000000000000000000000 P e2f3b5eb3fc34530b7e91cf02d665309 [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:09:15.922839 24088 raft_consensus.cc:513] T 00000000000000000000000000000000 P e2f3b5eb3fc34530b7e91cf02d665309 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2f3b5eb3fc34530b7e91cf02d665309" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 42775 } }
I20250903 08:09:15.923193 24088 leader_election.cc:304] T 00000000000000000000000000000000 P e2f3b5eb3fc34530b7e91cf02d665309 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e2f3b5eb3fc34530b7e91cf02d665309; no voters:
I20250903 08:09:15.923508 24088 leader_election.cc:290] T 00000000000000000000000000000000 P e2f3b5eb3fc34530b7e91cf02d665309 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250903 08:09:15.923882 24093 raft_consensus.cc:2802] T 00000000000000000000000000000000 P e2f3b5eb3fc34530b7e91cf02d665309 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 08:09:15.924062 24093 raft_consensus.cc:695] T 00000000000000000000000000000000 P e2f3b5eb3fc34530b7e91cf02d665309 [term 1 LEADER]: Becoming Leader. State: Replica: e2f3b5eb3fc34530b7e91cf02d665309, State: Running, Role: LEADER
I20250903 08:09:15.924335 24093 consensus_queue.cc:237] T 00000000000000000000000000000000 P e2f3b5eb3fc34530b7e91cf02d665309 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2f3b5eb3fc34530b7e91cf02d665309" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 42775 } }
I20250903 08:09:15.924729 24088 sys_catalog.cc:564] T 00000000000000000000000000000000 P e2f3b5eb3fc34530b7e91cf02d665309 [sys.catalog]: configured and running, proceeding with master startup.
I20250903 08:09:15.925840 24094 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2f3b5eb3fc34530b7e91cf02d665309 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e2f3b5eb3fc34530b7e91cf02d665309" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2f3b5eb3fc34530b7e91cf02d665309" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 42775 } } }
I20250903 08:09:15.925956 24094 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2f3b5eb3fc34530b7e91cf02d665309 [sys.catalog]: This master's current role is: LEADER
I20250903 08:09:15.926249 24095 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2f3b5eb3fc34530b7e91cf02d665309 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2f3b5eb3fc34530b7e91cf02d665309. Latest consensus state: current_term: 1 leader_uuid: "e2f3b5eb3fc34530b7e91cf02d665309" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2f3b5eb3fc34530b7e91cf02d665309" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 42775 } } }
I20250903 08:09:15.926332 24095 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2f3b5eb3fc34530b7e91cf02d665309 [sys.catalog]: This master's current role is: LEADER
I20250903 08:09:15.927052 24091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:09:15.914444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48066 (local address 127.12.201.254:42775)
0903 08:09:15.914855 (+ 411us) server_negotiation.cc:207] Beginning negotiation
0903 08:09:15.914865 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0903 08:09:15.914902 (+ 37us) server_negotiation.cc:408] Connection header received
0903 08:09:15.915542 (+ 640us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:09:15.915560 (+ 18us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:09:15.915924 (+ 364us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:09:15.916292 (+ 368us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:09:15.917335 (+ 1043us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:15.918111 (+ 776us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:09:15.918937 (+ 826us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:15.919189 (+ 252us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:15.921706 (+ 2517us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:09:15.921727 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:09:15.921737 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:09:15.921787 (+ 50us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:09:15.923724 (+ 1937us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:09:15.924232 (+ 508us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:09:15.924238 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:09:15.924244 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:09:15.924328 (+ 84us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:09:15.924621 (+ 293us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:09:15.924624 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:09:15.924626 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:09:15.925014 (+ 388us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:09:15.925217 (+ 203us) server_negotiation.cc:1036] Waiting for connection context
0903 08:09:15.925776 (+ 559us) server_negotiation.cc:300] Negotiation successful
0903 08:09:15.926170 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":133,"threads_started":1}
I20250903 08:09:15.927109 24102 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250903 08:09:15.930146 24102 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250903 08:09:15.935460 24102 catalog_manager.cc:1349] Generated new cluster ID: c178b7cb16c14ccea290e2ee98222632
I20250903 08:09:15.935547 24102 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250903 08:09:15.950950 24102 catalog_manager.cc:1372] Generated new certificate authority record
I20250903 08:09:15.951980 24102 catalog_manager.cc:1506] Loading token signing keys...
I20250903 08:09:15.958966 24102 catalog_manager.cc:5955] T 00000000000000000000000000000000 P e2f3b5eb3fc34530b7e91cf02d665309: Generated new TSK 0
I20250903 08:09:15.959725 24102 catalog_manager.cc:1516] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.201.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.201.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.201.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.201.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.201.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.201.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.201.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.201.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/krb5kdc/kudu.keytab.
I20250903 08:09:16.014195 13095 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUML90r/build/debug/bin/kudu
/tmp/dist-test-taskUML90r/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.201.193:0
--local_ip_for_outbound_sockets=127.12.201.193
--webserver_interface=127.12.201.193
--webserver_port=0
--tserver_master_addrs=127.12.201.254:42775
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.201.212:35551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 08:09:16.126191 24116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:09:16.126444 24116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:09:16.126507 24116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:09:16.129747 24116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 08:09:16.129810 24116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:09:16.129894 24116 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.201.193
I20250903 08:09:16.134023 24116 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.201.212:35551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.201.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.201.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.12.201.254:42775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24116
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.201.193
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 07:50:08 UTC on 5fd53c4cbb9d
build id 7874
I20250903 08:09:16.135056 24116 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:09:16.135864 24116 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:09:16.142017 24122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:09:16.142050 24124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:09:16.142019 24121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:09:16.142215 24116 server_base.cc:1047] running on GCE node
I20250903 08:09:16.142557 24116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:09:16.143136 24116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:09:16.144280 24116 hybrid_clock.cc:648] HybridClock initialized: now 1756886956144270 us; error 23 us; skew 500 ppm
Sep 03 08:09:16 dist-test-slave-27r9 krb5kdc[24004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886956, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 08:09:16.147030 24116 init.cc:377] Logged in from keytab as kudu/127.12.201.193@KRBTEST.COM (short username kudu)
I20250903 08:09:16.148015 24116 webserver.cc:480] Webserver started at http://127.12.201.193:40845/ using document root <none> and password file <none>
I20250903 08:09:16.148578 24116 fs_manager.cc:362] Metadata directory not provided
I20250903 08:09:16.148627 24116 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:09:16.148819 24116 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:09:16.150511 24116 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/minicluster-data/ts-0/data/instance:
uuid: "ca93febf097541379e21c45ca870266f"
format_stamp: "Formatted at 2025-09-03 08:09:16 on dist-test-slave-27r9"
server_key: "1bc8b5869b86ee51533fcb22df3e7ece"
server_key_iv: "468fbf4beeea94a084fa2dbff185abe0"
server_key_version: "encryptionkey@0"
I20250903 08:09:16.151057 24116 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/minicluster-data/ts-0/wal/instance:
uuid: "ca93febf097541379e21c45ca870266f"
format_stamp: "Formatted at 2025-09-03 08:09:16 on dist-test-slave-27r9"
server_key: "1bc8b5869b86ee51533fcb22df3e7ece"
server_key_iv: "468fbf4beeea94a084fa2dbff185abe0"
server_key_version: "encryptionkey@0"
I20250903 08:09:16.154404 24116 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20250903 08:09:16.156636 24131 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:09:16.157682 24116 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20250903 08:09:16.157794 24116 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/minicluster-data/ts-0/wal
uuid: "ca93febf097541379e21c45ca870266f"
format_stamp: "Formatted at 2025-09-03 08:09:16 on dist-test-slave-27r9"
server_key: "1bc8b5869b86ee51533fcb22df3e7ece"
server_key_iv: "468fbf4beeea94a084fa2dbff185abe0"
server_key_version: "encryptionkey@0"
I20250903 08:09:16.157881 24116 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:09:16.194578 24116 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:09:16.197402 24116 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:09:16.197598 24116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:09:16.198141 24116 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 08:09:16.199070 24116 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 08:09:16.199118 24116 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 08:09:16.199177 24116 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 08:09:16.199206 24116 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 08:09:16.208209 24116 rpc_server.cc:307] RPC server started. Bound to: 127.12.201.193:38163
I20250903 08:09:16.208250 24244 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.201.193:38163 every 8 connection(s)
I20250903 08:09:16.209128 24116 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/minicluster-data/ts-0/data/info.pb
I20250903 08:09:16.210238 13095 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUML90r/build/debug/bin/kudu as pid 24116
I20250903 08:09:16.210358 13095 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/minicluster-data/ts-0/wal/instance
I20250903 08:09:16.210600 13095 external_mini_cluster.cc:1468] Setting key 31e29facb1acc47b7915e108f51454e4
Sep 03 08:09:16 dist-test-slave-27r9 krb5kdc[24004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886956, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.193@KRBTEST.COM for kudu/127.12.201.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.201.194@KRBTEST.COM; defaulting to no policy
I20250903 08:09:16.222199 24091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:09:16.210792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.193:41825 (local address 127.12.201.254:42775)
0903 08:09:16.210962 (+ 170us) server_negotiation.cc:207] Beginning negotiation
0903 08:09:16.210966 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:09:16.211726 (+ 760us) server_negotiation.cc:408] Connection header received
0903 08:09:16.212786 (+ 1060us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:09:16.212789 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:09:16.212836 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:09:16.212905 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:09:16.214800 (+ 1895us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:16.215703 (+ 903us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:09:16.216369 (+ 666us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:16.216638 (+ 269us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:16.219012 (+ 2374us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:09:16.219031 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:09:16.219034 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:09:16.219058 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:09:16.220523 (+ 1465us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:09:16.221018 (+ 495us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:09:16.221021 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:09:16.221023 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:09:16.221065 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:09:16.221284 (+ 219us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:09:16.221286 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:09:16.221287 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:09:16.221419 (+ 132us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:09:16.221493 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
0903 08:09:16.221969 (+ 476us) server_negotiation.cc:300] Negotiation successful
0903 08:09:16.222074 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.201.194@KRBTEST.COM" created.
I20250903 08:09:16.222958 24247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:09:16.211134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:42775 (local address 127.12.201.193:41825)
0903 08:09:16.211591 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0903 08:09:16.211627 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0903 08:09:16.212567 (+ 940us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:09:16.213084 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:09:16.213097 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:09:16.213673 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:09:16.214636 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:09:16.214650 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:09:16.215853 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:09:16.215856 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:09:16.216247 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:09:16.216253 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:09:16.216420 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:16.217004 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:09:16.217028 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:09:16.218815 (+ 1787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:09:16.220646 (+ 1831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:09:16.220651 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:09:16.220661 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:09:16.220933 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:09:16.221139 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:09:16.221141 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:09:16.221142 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:09:16.221222 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:09:16.221492 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:09:16.221496 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:09:16.221750 (+ 254us) client_negotiation.cc:769] Sending connection context
0903 08:09:16.221935 (+ 185us) client_negotiation.cc:240] Negotiation successful
0903 08:09:16.222148 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":136,"threads_started":1}
I20250903 08:09:16.224123 24245 heartbeater.cc:344] Connected to a master server at 127.12.201.254:42775
I20250903 08:09:16.224354 24245 heartbeater.cc:461] Registering TS with master...
I20250903 08:09:16.224859 24245 heartbeater.cc:507] Master 127.12.201.254:42775 requested a full tablet report, sending...
I20250903 08:09:16.226255 24052 ts_manager.cc:194] Registered new tserver with Master: ca93febf097541379e21c45ca870266f (127.12.201.193:38163)
I20250903 08:09:16.227454 24052 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.201.193@KRBTEST.COM'} at 127.12.201.193:41825
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.201.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.201.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.201.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.201.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.201.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.201.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/krb5kdc/kudu.keytab.
I20250903 08:09:16.259934 13095 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUML90r/build/debug/bin/kudu
/tmp/dist-test-taskUML90r/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.201.194:0
--local_ip_for_outbound_sockets=127.12.201.194
--webserver_interface=127.12.201.194
--webserver_port=0
--tserver_master_addrs=127.12.201.254:42775
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.201.212:35551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 08:09:16.367209 24252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:09:16.367471 24252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:09:16.367563 24252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:09:16.370782 24252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 08:09:16.370854 24252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:09:16.370971 24252 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.201.194
I20250903 08:09:16.375200 24252 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.201.212:35551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.201.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.201.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.12.201.254:42775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24252
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.201.194
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 07:50:08 UTC on 5fd53c4cbb9d
build id 7874
I20250903 08:09:16.376271 24252 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:09:16.377133 24252 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:09:16.383502 24258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:09:16.383522 24257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:09:16.383745 24252 server_base.cc:1047] running on GCE node
W20250903 08:09:16.383754 24260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:09:16.384099 24252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:09:16.384639 24252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:09:16.385891 24252 hybrid_clock.cc:648] HybridClock initialized: now 1756886956385871 us; error 32 us; skew 500 ppm
Sep 03 08:09:16 dist-test-slave-27r9 krb5kdc[24004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886956, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 08:09:16.388577 24252 init.cc:377] Logged in from keytab as kudu/127.12.201.194@KRBTEST.COM (short username kudu)
I20250903 08:09:16.389576 24252 webserver.cc:480] Webserver started at http://127.12.201.194:41403/ using document root <none> and password file <none>
I20250903 08:09:16.390106 24252 fs_manager.cc:362] Metadata directory not provided
I20250903 08:09:16.390158 24252 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:09:16.390336 24252 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:09:16.392102 24252 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/minicluster-data/ts-1/data/instance:
uuid: "1c58657070824b90b752c6cd04a89eb1"
format_stamp: "Formatted at 2025-09-03 08:09:16 on dist-test-slave-27r9"
server_key: "c6223c0df5185a3b29d089020d98b8a9"
server_key_iv: "6b2d2a2c739ba3f0b48379e1bcf9738a"
server_key_version: "encryptionkey@0"
I20250903 08:09:16.392581 24252 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/minicluster-data/ts-1/wal/instance:
uuid: "1c58657070824b90b752c6cd04a89eb1"
format_stamp: "Formatted at 2025-09-03 08:09:16 on dist-test-slave-27r9"
server_key: "c6223c0df5185a3b29d089020d98b8a9"
server_key_iv: "6b2d2a2c739ba3f0b48379e1bcf9738a"
server_key_version: "encryptionkey@0"
I20250903 08:09:16.395985 24252 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20250903 08:09:16.398043 24267 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:09:16.399089 24252 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250903 08:09:16.399196 24252 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/minicluster-data/ts-1/wal
uuid: "1c58657070824b90b752c6cd04a89eb1"
format_stamp: "Formatted at 2025-09-03 08:09:16 on dist-test-slave-27r9"
server_key: "c6223c0df5185a3b29d089020d98b8a9"
server_key_iv: "6b2d2a2c739ba3f0b48379e1bcf9738a"
server_key_version: "encryptionkey@0"
I20250903 08:09:16.399282 24252 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:09:16.414144 24252 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:09:16.416872 24252 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:09:16.417040 24252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:09:16.417547 24252 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 08:09:16.418361 24252 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 08:09:16.418408 24252 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 08:09:16.418469 24252 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 08:09:16.418496 24252 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 08:09:16.427577 24252 rpc_server.cc:307] RPC server started. Bound to: 127.12.201.194:37993
I20250903 08:09:16.427636 24380 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.201.194:37993 every 8 connection(s)
I20250903 08:09:16.428568 24252 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/minicluster-data/ts-1/data/info.pb
I20250903 08:09:16.435786 13095 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUML90r/build/debug/bin/kudu as pid 24252
I20250903 08:09:16.435904 13095 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/minicluster-data/ts-1/wal/instance
I20250903 08:09:16.436137 13095 external_mini_cluster.cc:1468] Setting key ec081627df32701103faa32827b29283
Sep 03 08:09:16 dist-test-slave-27r9 krb5kdc[24004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886956, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.194@KRBTEST.COM for kudu/127.12.201.254@KRBTEST.COM
I20250903 08:09:16.440846 24091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:09:16.430221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.194:39033 (local address 127.12.201.254:42775)
0903 08:09:16.430343 (+ 122us) server_negotiation.cc:207] Beginning negotiation
0903 08:09:16.430346 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0903 08:09:16.431049 (+ 703us) server_negotiation.cc:408] Connection header received
0903 08:09:16.431903 (+ 854us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:09:16.431906 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:09:16.431955 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:09:16.432031 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:09:16.433588 (+ 1557us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:16.434103 (+ 515us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:09:16.434759 (+ 656us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:16.434910 (+ 151us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:16.437186 (+ 2276us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:09:16.437205 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:09:16.437208 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:09:16.437234 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:09:16.438773 (+ 1539us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:09:16.439404 (+ 631us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:09:16.439409 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:09:16.439412 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:09:16.439516 (+ 104us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:09:16.439810 (+ 294us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:09:16.439813 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:09:16.439815 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:09:16.439969 (+ 154us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:09:16.440048 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
0903 08:09:16.440540 (+ 492us) server_negotiation.cc:300] Negotiation successful
0903 08:09:16.440674 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20250903 08:09:16.441514 24383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:09:16.430488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:42775 (local address 127.12.201.194:39033)
0903 08:09:16.430909 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0903 08:09:16.430945 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0903 08:09:16.431707 (+ 762us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:09:16.432163 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:09:16.432171 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:09:16.432569 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:09:16.433462 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:09:16.433477 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:09:16.434221 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:09:16.434224 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:09:16.434634 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:09:16.434640 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:09:16.434810 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:16.435414 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:09:16.435438 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:09:16.437041 (+ 1603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:09:16.438892 (+ 1851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:09:16.438902 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:09:16.438919 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:09:16.439305 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:09:16.439630 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:09:16.439633 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:09:16.439634 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:09:16.439720 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:09:16.440045 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:09:16.440050 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:09:16.440275 (+ 225us) client_negotiation.cc:769] Sending connection context
0903 08:09:16.440467 (+ 192us) client_negotiation.cc:240] Negotiation successful
0903 08:09:16.440739 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":108,"threads_started":1}
I20250903 08:09:16.442744 24381 heartbeater.cc:344] Connected to a master server at 127.12.201.254:42775
I20250903 08:09:16.442972 24381 heartbeater.cc:461] Registering TS with master...
I20250903 08:09:16.443529 24381 heartbeater.cc:507] Master 127.12.201.254:42775 requested a full tablet report, sending...
I20250903 08:09:16.444573 24052 ts_manager.cc:194] Registered new tserver with Master: 1c58657070824b90b752c6cd04a89eb1 (127.12.201.194:37993)
WARNING: no policy specified for kudu/127.12.201.195@KRBTEST.COM; defaulting to no policy
I20250903 08:09:16.445384 24052 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.201.194@KRBTEST.COM'} at 127.12.201.194:39033
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.201.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.201.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.201.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.201.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.201.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.201.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.201.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/krb5kdc/kudu.keytab.
I20250903 08:09:16.484437 13095 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUML90r/build/debug/bin/kudu
/tmp/dist-test-taskUML90r/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.201.195:0
--local_ip_for_outbound_sockets=127.12.201.195
--webserver_interface=127.12.201.195
--webserver_port=0
--tserver_master_addrs=127.12.201.254:42775
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.201.212:35551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 08:09:16.591225 24388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:09:16.591455 24388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:09:16.591538 24388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:09:16.594732 24388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 08:09:16.594801 24388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:09:16.594903 24388 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.201.195
I20250903 08:09:16.599156 24388 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.201.212:35551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.201.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.201.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.12.201.254:42775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24388
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.201.195
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 07:50:08 UTC on 5fd53c4cbb9d
build id 7874
I20250903 08:09:16.600257 24388 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:09:16.601130 24388 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:09:16.607739 24393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:09:16.607703 24396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:09:16.607703 24394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:09:16.608050 24388 server_base.cc:1047] running on GCE node
I20250903 08:09:16.608428 24388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:09:16.608985 24388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:09:16.610137 24388 hybrid_clock.cc:648] HybridClock initialized: now 1756886956610127 us; error 32 us; skew 500 ppm
Sep 03 08:09:16 dist-test-slave-27r9 krb5kdc[24004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886956, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 08:09:16.612947 24388 init.cc:377] Logged in from keytab as kudu/127.12.201.195@KRBTEST.COM (short username kudu)
I20250903 08:09:16.614028 24388 webserver.cc:480] Webserver started at http://127.12.201.195:41735/ using document root <none> and password file <none>
I20250903 08:09:16.614560 24388 fs_manager.cc:362] Metadata directory not provided
I20250903 08:09:16.614606 24388 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:09:16.614773 24388 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:09:16.616578 24388 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/minicluster-data/ts-2/data/instance:
uuid: "3c296b676ee9444394e0b09933dd3545"
format_stamp: "Formatted at 2025-09-03 08:09:16 on dist-test-slave-27r9"
server_key: "7e128b2f6f374718598b44ca9ef1a876"
server_key_iv: "76ef2b74c09dee8eb8a25cc155b3eae7"
server_key_version: "encryptionkey@0"
I20250903 08:09:16.617014 24388 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/minicluster-data/ts-2/wal/instance:
uuid: "3c296b676ee9444394e0b09933dd3545"
format_stamp: "Formatted at 2025-09-03 08:09:16 on dist-test-slave-27r9"
server_key: "7e128b2f6f374718598b44ca9ef1a876"
server_key_iv: "76ef2b74c09dee8eb8a25cc155b3eae7"
server_key_version: "encryptionkey@0"
I20250903 08:09:16.620507 24388 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250903 08:09:16.622839 24403 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:09:16.624001 24388 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250903 08:09:16.624110 24388 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/minicluster-data/ts-2/wal
uuid: "3c296b676ee9444394e0b09933dd3545"
format_stamp: "Formatted at 2025-09-03 08:09:16 on dist-test-slave-27r9"
server_key: "7e128b2f6f374718598b44ca9ef1a876"
server_key_iv: "76ef2b74c09dee8eb8a25cc155b3eae7"
server_key_version: "encryptionkey@0"
I20250903 08:09:16.624233 24388 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:09:16.636991 24388 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:09:16.640054 24388 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:09:16.640244 24388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:09:16.640811 24388 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 08:09:16.641664 24388 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 08:09:16.641710 24388 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 08:09:16.641764 24388 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 08:09:16.641799 24388 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 08:09:16.651145 24388 rpc_server.cc:307] RPC server started. Bound to: 127.12.201.195:43221
I20250903 08:09:16.651173 24516 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.201.195:43221 every 8 connection(s)
I20250903 08:09:16.652113 24388 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/minicluster-data/ts-2/data/info.pb
I20250903 08:09:16.659842 13095 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUML90r/build/debug/bin/kudu as pid 24388
I20250903 08:09:16.659997 13095 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/minicluster-data/ts-2/wal/instance
I20250903 08:09:16.660260 13095 external_mini_cluster.cc:1468] Setting key 5438a105451d6d3273a16ee0b4db825c
Sep 03 08:09:16 dist-test-slave-27r9 krb5kdc[24004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886956, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.195@KRBTEST.COM for kudu/127.12.201.254@KRBTEST.COM
I20250903 08:09:16.664605 24091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:09:16.653830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.195:57423 (local address 127.12.201.254:42775)
0903 08:09:16.654002 (+ 172us) server_negotiation.cc:207] Beginning negotiation
0903 08:09:16.654007 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 08:09:16.654711 (+ 704us) server_negotiation.cc:408] Connection header received
0903 08:09:16.655594 (+ 883us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:09:16.655597 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:09:16.655647 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:09:16.655724 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:09:16.657502 (+ 1778us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:16.658011 (+ 509us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:09:16.658588 (+ 577us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:16.658713 (+ 125us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:16.661223 (+ 2510us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:09:16.661239 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:09:16.661241 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:09:16.661266 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:09:16.662758 (+ 1492us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:09:16.663279 (+ 521us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:09:16.663282 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:09:16.663284 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:09:16.663328 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:09:16.663616 (+ 288us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:09:16.663619 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:09:16.663620 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:09:16.663783 (+ 163us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:09:16.663868 (+ 85us) server_negotiation.cc:1036] Waiting for connection context
0903 08:09:16.664362 (+ 494us) server_negotiation.cc:300] Negotiation successful
0903 08:09:16.664478 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20250903 08:09:16.665258 24519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:09:16.654129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:42775 (local address 127.12.201.195:57423)
0903 08:09:16.654566 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0903 08:09:16.654604 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0903 08:09:16.655399 (+ 795us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:09:16.655882 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:09:16.655891 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:09:16.656315 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:09:16.657343 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:09:16.657360 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:09:16.658115 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:09:16.658118 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:09:16.658505 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:09:16.658512 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:09:16.658666 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:16.659308 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:09:16.659333 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:09:16.661044 (+ 1711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:09:16.662874 (+ 1830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:09:16.662879 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:09:16.662889 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:09:16.663186 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:09:16.663399 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:09:16.663401 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:09:16.663403 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:09:16.663496 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:09:16.663870 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:09:16.663874 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:09:16.664110 (+ 236us) client_negotiation.cc:769] Sending connection context
0903 08:09:16.664282 (+ 172us) client_negotiation.cc:240] Negotiation successful
0903 08:09:16.664522 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":110,"threads_started":1}
I20250903 08:09:16.666388 24517 heartbeater.cc:344] Connected to a master server at 127.12.201.254:42775
I20250903 08:09:16.666617 24517 heartbeater.cc:461] Registering TS with master...
I20250903 08:09:16.667171 24517 heartbeater.cc:507] Master 127.12.201.254:42775 requested a full tablet report, sending...
I20250903 08:09:16.668211 24052 ts_manager.cc:194] Registered new tserver with Master: 3c296b676ee9444394e0b09933dd3545 (127.12.201.195:43221)
I20250903 08:09:16.668817 24052 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.201.195@KRBTEST.COM'} at 127.12.201.195:57423
I20250903 08:09:16.673944 13095 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250903 08:09:16.682355 24091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:09:16.675860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48082 (local address 127.12.201.254:42775)
0903 08:09:16.675976 (+ 116us) server_negotiation.cc:207] Beginning negotiation
0903 08:09:16.675980 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:09:16.676073 (+ 93us) server_negotiation.cc:408] Connection header received
0903 08:09:16.676176 (+ 103us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:09:16.676178 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:09:16.676222 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:09:16.676298 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:09:16.677096 (+ 798us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:16.677697 (+ 601us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:09:16.678315 (+ 618us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:16.678451 (+ 136us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:16.679641 (+ 1190us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:09:16.679657 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:09:16.679659 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:09:16.679684 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:09:16.681087 (+ 1403us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:09:16.681477 (+ 390us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:09:16.681480 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:09:16.681481 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:09:16.681523 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:09:16.681721 (+ 198us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:09:16.681723 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:09:16.681724 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:09:16.681887 (+ 163us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:09:16.681985 (+ 98us) server_negotiation.cc:1036] Waiting for connection context
0903 08:09:16.682144 (+ 159us) server_negotiation.cc:300] Negotiation successful
0903 08:09:16.682232 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20250903 08:09:16.685091 24052 catalog_manager.cc:2232] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48082:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 08:09:16.687532 24052 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 08:09:16.699237 24528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:09:16.695314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.195:43221 (local address 127.0.0.1:54184)
0903 08:09:16.695666 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0903 08:09:16.695714 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0903 08:09:16.695842 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:09:16.696475 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:09:16.696484 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:09:16.696518 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 08:09:16.696831 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:09:16.696839 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:09:16.698090 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:09:16.698094 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:09:16.698675 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:09:16.698683 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:09:16.698836 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:16.698886 (+ 50us) client_negotiation.cc:769] Sending connection context
0903 08:09:16.698969 (+ 83us) client_negotiation.cc:240] Negotiation successful
0903 08:09:16.699095 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":91,"threads_started":1}
I20250903 08:09:16.699839 24529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:09:16.695433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54184 (local address 127.12.201.195:43221)
0903 08:09:16.695882 (+ 449us) server_negotiation.cc:207] Beginning negotiation
0903 08:09:16.695889 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0903 08:09:16.695914 (+ 25us) server_negotiation.cc:408] Connection header received
0903 08:09:16.695998 (+ 84us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:09:16.696004 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:09:16.696189 (+ 185us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:09:16.696321 (+ 132us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 08:09:16.696959 (+ 638us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:16.697993 (+ 1034us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:09:16.698871 (+ 878us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:16.699401 (+ 530us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:16.699528 (+ 127us) server_negotiation.cc:1036] Waiting for connection context
0903 08:09:16.699597 (+ 69us) server_negotiation.cc:300] Negotiation successful
0903 08:09:16.699689 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":103,"threads_started":1}
I20250903 08:09:16.700044 24531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:09:16.695999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.194:37993 (local address 127.0.0.1:37638)
0903 08:09:16.696609 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0903 08:09:16.696624 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0903 08:09:16.696699 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:09:16.697235 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:09:16.697238 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:09:16.697255 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 08:09:16.697478 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:09:16.697486 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:09:16.698758 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:09:16.698763 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:09:16.699667 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:09:16.699680 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:09:16.699793 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:16.699814 (+ 21us) client_negotiation.cc:769] Sending connection context
0903 08:09:16.699868 (+ 54us) client_negotiation.cc:240] Negotiation successful
0903 08:09:16.699926 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":110,"threads_started":1}
I20250903 08:09:16.700784 24532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:09:16.696167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37638 (local address 127.12.201.194:37993)
0903 08:09:16.696839 (+ 672us) server_negotiation.cc:207] Beginning negotiation
0903 08:09:16.696846 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0903 08:09:16.696868 (+ 22us) server_negotiation.cc:408] Connection header received
0903 08:09:16.696948 (+ 80us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:09:16.696953 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:09:16.697104 (+ 151us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:09:16.697237 (+ 133us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 08:09:16.697594 (+ 357us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:16.698631 (+ 1037us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:09:16.699853 (+ 1222us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:16.700344 (+ 491us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:16.700457 (+ 113us) server_negotiation.cc:1036] Waiting for connection context
0903 08:09:16.700559 (+ 102us) server_negotiation.cc:300] Negotiation successful
0903 08:09:16.700645 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":94,"threads_started":1}
I20250903 08:09:16.701084 24530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:09:16.695520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.193:38163 (local address 127.0.0.1:51974)
0903 08:09:16.696095 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0903 08:09:16.696108 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0903 08:09:16.696183 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:09:16.697474 (+ 1291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:09:16.697478 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:09:16.697496 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 08:09:16.697731 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:09:16.697738 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:09:16.699846 (+ 2108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:09:16.699852 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:09:16.700732 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:09:16.700743 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:09:16.700851 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:16.700867 (+ 16us) client_negotiation.cc:769] Sending connection context
0903 08:09:16.700915 (+ 48us) client_negotiation.cc:240] Negotiation successful
0903 08:09:16.700969 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":86,"threads_started":1}
I20250903 08:09:16.701772 24533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:09:16.696301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51974 (local address 127.12.201.193:38163)
0903 08:09:16.696940 (+ 639us) server_negotiation.cc:207] Beginning negotiation
0903 08:09:16.696947 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0903 08:09:16.696968 (+ 21us) server_negotiation.cc:408] Connection header received
0903 08:09:16.697052 (+ 84us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:09:16.697057 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:09:16.697210 (+ 153us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:09:16.697330 (+ 120us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 08:09:16.697854 (+ 524us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:16.698935 (+ 1081us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:09:16.700904 (+ 1969us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:16.701408 (+ 504us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:16.701503 (+ 95us) server_negotiation.cc:1036] Waiting for connection context
0903 08:09:16.701568 (+ 65us) server_negotiation.cc:300] Negotiation successful
0903 08:09:16.701661 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":118,"threads_started":1}
I20250903 08:09:16.702678 24451 tablet_service.cc:1468] Processing CreateTablet for tablet adcd006723944e0eb7983a66b1e8e70d (DEFAULT_TABLE table=test-table [id=9b0a57e5a93649319dccbd42e6338931]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:09:16.703197 24315 tablet_service.cc:1468] Processing CreateTablet for tablet adcd006723944e0eb7983a66b1e8e70d (DEFAULT_TABLE table=test-table [id=9b0a57e5a93649319dccbd42e6338931]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:09:16.703716 24451 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet adcd006723944e0eb7983a66b1e8e70d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:09:16.703794 24179 tablet_service.cc:1468] Processing CreateTablet for tablet adcd006723944e0eb7983a66b1e8e70d (DEFAULT_TABLE table=test-table [id=9b0a57e5a93649319dccbd42e6338931]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:09:16.704052 24315 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet adcd006723944e0eb7983a66b1e8e70d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:09:16.704478 24179 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet adcd006723944e0eb7983a66b1e8e70d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:09:16.709506 24534 tablet_bootstrap.cc:492] T adcd006723944e0eb7983a66b1e8e70d P ca93febf097541379e21c45ca870266f: Bootstrap starting.
I20250903 08:09:16.710565 24535 tablet_bootstrap.cc:492] T adcd006723944e0eb7983a66b1e8e70d P 1c58657070824b90b752c6cd04a89eb1: Bootstrap starting.
I20250903 08:09:16.710721 24536 tablet_bootstrap.cc:492] T adcd006723944e0eb7983a66b1e8e70d P 3c296b676ee9444394e0b09933dd3545: Bootstrap starting.
I20250903 08:09:16.711735 24534 tablet_bootstrap.cc:654] T adcd006723944e0eb7983a66b1e8e70d P ca93febf097541379e21c45ca870266f: Neither blocks nor log segments found. Creating new log.
I20250903 08:09:16.712430 24534 log.cc:826] T adcd006723944e0eb7983a66b1e8e70d P ca93febf097541379e21c45ca870266f: Log is configured to *not* fsync() on all Append() calls
I20250903 08:09:16.712661 24536 tablet_bootstrap.cc:654] T adcd006723944e0eb7983a66b1e8e70d P 3c296b676ee9444394e0b09933dd3545: Neither blocks nor log segments found. Creating new log.
I20250903 08:09:16.712661 24535 tablet_bootstrap.cc:654] T adcd006723944e0eb7983a66b1e8e70d P 1c58657070824b90b752c6cd04a89eb1: Neither blocks nor log segments found. Creating new log.
I20250903 08:09:16.713469 24535 log.cc:826] T adcd006723944e0eb7983a66b1e8e70d P 1c58657070824b90b752c6cd04a89eb1: Log is configured to *not* fsync() on all Append() calls
I20250903 08:09:16.713471 24536 log.cc:826] T adcd006723944e0eb7983a66b1e8e70d P 3c296b676ee9444394e0b09933dd3545: Log is configured to *not* fsync() on all Append() calls
I20250903 08:09:16.714213 24534 tablet_bootstrap.cc:492] T adcd006723944e0eb7983a66b1e8e70d P ca93febf097541379e21c45ca870266f: No bootstrap required, opened a new log
I20250903 08:09:16.714417 24534 ts_tablet_manager.cc:1397] T adcd006723944e0eb7983a66b1e8e70d P ca93febf097541379e21c45ca870266f: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20250903 08:09:16.715292 24536 tablet_bootstrap.cc:492] T adcd006723944e0eb7983a66b1e8e70d P 3c296b676ee9444394e0b09933dd3545: No bootstrap required, opened a new log
I20250903 08:09:16.715490 24536 ts_tablet_manager.cc:1397] T adcd006723944e0eb7983a66b1e8e70d P 3c296b676ee9444394e0b09933dd3545: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250903 08:09:16.715591 24535 tablet_bootstrap.cc:492] T adcd006723944e0eb7983a66b1e8e70d P 1c58657070824b90b752c6cd04a89eb1: No bootstrap required, opened a new log
I20250903 08:09:16.715776 24535 ts_tablet_manager.cc:1397] T adcd006723944e0eb7983a66b1e8e70d P 1c58657070824b90b752c6cd04a89eb1: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250903 08:09:16.717140 24534 raft_consensus.cc:357] T adcd006723944e0eb7983a66b1e8e70d P ca93febf097541379e21c45ca870266f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c296b676ee9444394e0b09933dd3545" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 43221 } } peers { permanent_uuid: "1c58657070824b90b752c6cd04a89eb1" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 37993 } } peers { permanent_uuid: "ca93febf097541379e21c45ca870266f" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 38163 } }
I20250903 08:09:16.717360 24534 raft_consensus.cc:383] T adcd006723944e0eb7983a66b1e8e70d P ca93febf097541379e21c45ca870266f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:09:16.717437 24534 raft_consensus.cc:738] T adcd006723944e0eb7983a66b1e8e70d P ca93febf097541379e21c45ca870266f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca93febf097541379e21c45ca870266f, State: Initialized, Role: FOLLOWER
I20250903 08:09:16.717875 24534 consensus_queue.cc:260] T adcd006723944e0eb7983a66b1e8e70d P ca93febf097541379e21c45ca870266f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c296b676ee9444394e0b09933dd3545" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 43221 } } peers { permanent_uuid: "1c58657070824b90b752c6cd04a89eb1" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 37993 } } peers { permanent_uuid: "ca93febf097541379e21c45ca870266f" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 38163 } }
I20250903 08:09:16.718541 24245 heartbeater.cc:499] Master 127.12.201.254:42775 was elected leader, sending a full tablet report...
I20250903 08:09:16.718710 24536 raft_consensus.cc:357] T adcd006723944e0eb7983a66b1e8e70d P 3c296b676ee9444394e0b09933dd3545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c296b676ee9444394e0b09933dd3545" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 43221 } } peers { permanent_uuid: "1c58657070824b90b752c6cd04a89eb1" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 37993 } } peers { permanent_uuid: "ca93febf097541379e21c45ca870266f" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 38163 } }
I20250903 08:09:16.718907 24534 ts_tablet_manager.cc:1428] T adcd006723944e0eb7983a66b1e8e70d P ca93febf097541379e21c45ca870266f: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20250903 08:09:16.718915 24536 raft_consensus.cc:383] T adcd006723944e0eb7983a66b1e8e70d P 3c296b676ee9444394e0b09933dd3545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:09:16.718967 24536 raft_consensus.cc:738] T adcd006723944e0eb7983a66b1e8e70d P 3c296b676ee9444394e0b09933dd3545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c296b676ee9444394e0b09933dd3545, State: Initialized, Role: FOLLOWER
I20250903 08:09:16.719439 24536 consensus_queue.cc:260] T adcd006723944e0eb7983a66b1e8e70d P 3c296b676ee9444394e0b09933dd3545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c296b676ee9444394e0b09933dd3545" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 43221 } } peers { permanent_uuid: "1c58657070824b90b752c6cd04a89eb1" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 37993 } } peers { permanent_uuid: "ca93febf097541379e21c45ca870266f" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 38163 } }
I20250903 08:09:16.719889 24535 raft_consensus.cc:357] T adcd006723944e0eb7983a66b1e8e70d P 1c58657070824b90b752c6cd04a89eb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c296b676ee9444394e0b09933dd3545" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 43221 } } peers { permanent_uuid: "1c58657070824b90b752c6cd04a89eb1" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 37993 } } peers { permanent_uuid: "ca93febf097541379e21c45ca870266f" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 38163 } }
I20250903 08:09:16.720110 24517 heartbeater.cc:499] Master 127.12.201.254:42775 was elected leader, sending a full tablet report...
I20250903 08:09:16.720137 24535 raft_consensus.cc:383] T adcd006723944e0eb7983a66b1e8e70d P 1c58657070824b90b752c6cd04a89eb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:09:16.720271 24535 raft_consensus.cc:738] T adcd006723944e0eb7983a66b1e8e70d P 1c58657070824b90b752c6cd04a89eb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c58657070824b90b752c6cd04a89eb1, State: Initialized, Role: FOLLOWER
I20250903 08:09:16.720389 24536 ts_tablet_manager.cc:1428] T adcd006723944e0eb7983a66b1e8e70d P 3c296b676ee9444394e0b09933dd3545: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20250903 08:09:16.720759 24535 consensus_queue.cc:260] T adcd006723944e0eb7983a66b1e8e70d P 1c58657070824b90b752c6cd04a89eb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c296b676ee9444394e0b09933dd3545" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 43221 } } peers { permanent_uuid: "1c58657070824b90b752c6cd04a89eb1" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 37993 } } peers { permanent_uuid: "ca93febf097541379e21c45ca870266f" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 38163 } }
I20250903 08:09:16.721329 24381 heartbeater.cc:499] Master 127.12.201.254:42775 was elected leader, sending a full tablet report...
I20250903 08:09:16.721549 24535 ts_tablet_manager.cc:1428] T adcd006723944e0eb7983a66b1e8e70d P 1c58657070824b90b752c6cd04a89eb1: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20250903 08:09:16.730664 24540 raft_consensus.cc:491] T adcd006723944e0eb7983a66b1e8e70d P ca93febf097541379e21c45ca870266f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 08:09:16.730861 24540 raft_consensus.cc:513] T adcd006723944e0eb7983a66b1e8e70d P ca93febf097541379e21c45ca870266f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c296b676ee9444394e0b09933dd3545" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 43221 } } peers { permanent_uuid: "1c58657070824b90b752c6cd04a89eb1" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 37993 } } peers { permanent_uuid: "ca93febf097541379e21c45ca870266f" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 38163 } }
I20250903 08:09:16.732065 24540 leader_election.cc:290] T adcd006723944e0eb7983a66b1e8e70d P ca93febf097541379e21c45ca870266f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c296b676ee9444394e0b09933dd3545 (127.12.201.195:43221), 1c58657070824b90b752c6cd04a89eb1 (127.12.201.194:37993)
I20250903 08:09:16.735661 24544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:09:16.732617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.194:37993 (local address 127.12.201.193:58661)
0903 08:09:16.732905 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0903 08:09:16.732914 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0903 08:09:16.733020 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:09:16.733264 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:09:16.733267 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:09:16.733281 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 08:09:16.733535 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:09:16.733540 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:09:16.734434 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:09:16.734437 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:09:16.735259 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:09:16.735268 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:09:16.735354 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:16.735364 (+ 10us) client_negotiation.cc:769] Sending connection context
0903 08:09:16.735413 (+ 49us) client_negotiation.cc:240] Negotiation successful
0903 08:09:16.735473 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"spinlock_wait_cycles":6784,"thread_start_us":108,"threads_started":1}
I20250903 08:09:16.735663 24543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:09:16.732236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.195:43221 (local address 127.12.201.193:56675)
0903 08:09:16.732488 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0903 08:09:16.732504 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0903 08:09:16.732609 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:09:16.732979 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:09:16.732982 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:09:16.733017 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 08:09:16.733328 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:09:16.733336 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:09:16.734297 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:09:16.734299 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:09:16.735193 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:09:16.735203 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:09:16.735328 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:16.735352 (+ 24us) client_negotiation.cc:769] Sending connection context
0903 08:09:16.735408 (+ 56us) client_negotiation.cc:240] Negotiation successful
0903 08:09:16.735482 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"spinlock_wait_cycles":4608,"thread_start_us":102,"threads_started":1}
I20250903 08:09:16.736094 24532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:09:16.732783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.193:58661 (local address 127.12.201.194:37993)
0903 08:09:16.732906 (+ 123us) server_negotiation.cc:207] Beginning negotiation
0903 08:09:16.732912 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0903 08:09:16.732969 (+ 57us) server_negotiation.cc:408] Connection header received
0903 08:09:16.733118 (+ 149us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:09:16.733120 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:09:16.733177 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:09:16.733238 (+ 61us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 08:09:16.733640 (+ 402us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:16.734349 (+ 709us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:09:16.735360 (+ 1011us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:16.735826 (+ 466us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:16.735865 (+ 39us) server_negotiation.cc:1036] Waiting for connection context
0903 08:09:16.735922 (+ 57us) server_negotiation.cc:300] Negotiation successful
0903 08:09:16.735978 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20250903 08:09:16.736094 24529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:09:16.732297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.193:56675 (local address 127.12.201.195:43221)
0903 08:09:16.732460 (+ 163us) server_negotiation.cc:207] Beginning negotiation
0903 08:09:16.732465 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 08:09:16.732584 (+ 119us) server_negotiation.cc:408] Connection header received
0903 08:09:16.732807 (+ 223us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:09:16.732810 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:09:16.732867 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:09:16.732954 (+ 87us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 08:09:16.733464 (+ 510us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:16.734185 (+ 721us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:09:16.735308 (+ 1123us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:16.735826 (+ 518us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:16.735865 (+ 39us) server_negotiation.cc:1036] Waiting for connection context
0903 08:09:16.735922 (+ 57us) server_negotiation.cc:300] Negotiation successful
0903 08:09:16.735978 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20250903 08:09:16.736748 24335 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "adcd006723944e0eb7983a66b1e8e70d" candidate_uuid: "ca93febf097541379e21c45ca870266f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c58657070824b90b752c6cd04a89eb1" is_pre_election: true
I20250903 08:09:16.736721 24471 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "adcd006723944e0eb7983a66b1e8e70d" candidate_uuid: "ca93febf097541379e21c45ca870266f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c296b676ee9444394e0b09933dd3545" is_pre_election: true
I20250903 08:09:16.737028 24335 raft_consensus.cc:2466] T adcd006723944e0eb7983a66b1e8e70d P 1c58657070824b90b752c6cd04a89eb1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca93febf097541379e21c45ca870266f in term 0.
I20250903 08:09:16.737028 24471 raft_consensus.cc:2466] T adcd006723944e0eb7983a66b1e8e70d P 3c296b676ee9444394e0b09933dd3545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca93febf097541379e21c45ca870266f in term 0.
I20250903 08:09:16.737516 24132 leader_election.cc:304] T adcd006723944e0eb7983a66b1e8e70d P ca93febf097541379e21c45ca870266f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c296b676ee9444394e0b09933dd3545, ca93febf097541379e21c45ca870266f; no voters:
I20250903 08:09:16.737759 24540 raft_consensus.cc:2802] T adcd006723944e0eb7983a66b1e8e70d P ca93febf097541379e21c45ca870266f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 08:09:16.737836 24540 raft_consensus.cc:491] T adcd006723944e0eb7983a66b1e8e70d P ca93febf097541379e21c45ca870266f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 08:09:16.737883 24540 raft_consensus.cc:3058] T adcd006723944e0eb7983a66b1e8e70d P ca93febf097541379e21c45ca870266f [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:09:16.738814 24540 raft_consensus.cc:513] T adcd006723944e0eb7983a66b1e8e70d P ca93febf097541379e21c45ca870266f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c296b676ee9444394e0b09933dd3545" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 43221 } } peers { permanent_uuid: "1c58657070824b90b752c6cd04a89eb1" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 37993 } } peers { permanent_uuid: "ca93febf097541379e21c45ca870266f" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 38163 } }
I20250903 08:09:16.739180 24540 leader_election.cc:290] T adcd006723944e0eb7983a66b1e8e70d P ca93febf097541379e21c45ca870266f [CANDIDATE]: Term 1 election: Requested vote from peers 3c296b676ee9444394e0b09933dd3545 (127.12.201.195:43221), 1c58657070824b90b752c6cd04a89eb1 (127.12.201.194:37993)
I20250903 08:09:16.739606 24471 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "adcd006723944e0eb7983a66b1e8e70d" candidate_uuid: "ca93febf097541379e21c45ca870266f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c296b676ee9444394e0b09933dd3545"
I20250903 08:09:16.739733 24471 raft_consensus.cc:3058] T adcd006723944e0eb7983a66b1e8e70d P 3c296b676ee9444394e0b09933dd3545 [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:09:16.739780 24335 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "adcd006723944e0eb7983a66b1e8e70d" candidate_uuid: "ca93febf097541379e21c45ca870266f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c58657070824b90b752c6cd04a89eb1"
I20250903 08:09:16.739902 24335 raft_consensus.cc:3058] T adcd006723944e0eb7983a66b1e8e70d P 1c58657070824b90b752c6cd04a89eb1 [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:09:16.740974 24471 raft_consensus.cc:2466] T adcd006723944e0eb7983a66b1e8e70d P 3c296b676ee9444394e0b09933dd3545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca93febf097541379e21c45ca870266f in term 1.
I20250903 08:09:16.741016 24335 raft_consensus.cc:2466] T adcd006723944e0eb7983a66b1e8e70d P 1c58657070824b90b752c6cd04a89eb1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca93febf097541379e21c45ca870266f in term 1.
I20250903 08:09:16.741299 24132 leader_election.cc:304] T adcd006723944e0eb7983a66b1e8e70d P ca93febf097541379e21c45ca870266f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c296b676ee9444394e0b09933dd3545, ca93febf097541379e21c45ca870266f; no voters:
I20250903 08:09:16.741454 24540 raft_consensus.cc:2802] T adcd006723944e0eb7983a66b1e8e70d P ca93febf097541379e21c45ca870266f [term 1 FOLLOWER]: Leader election won for term 1
I20250903 08:09:16.741706 24540 raft_consensus.cc:695] T adcd006723944e0eb7983a66b1e8e70d P ca93febf097541379e21c45ca870266f [term 1 LEADER]: Becoming Leader. State: Replica: ca93febf097541379e21c45ca870266f, State: Running, Role: LEADER
I20250903 08:09:16.742053 24540 consensus_queue.cc:237] T adcd006723944e0eb7983a66b1e8e70d P ca93febf097541379e21c45ca870266f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c296b676ee9444394e0b09933dd3545" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 43221 } } peers { permanent_uuid: "1c58657070824b90b752c6cd04a89eb1" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 37993 } } peers { permanent_uuid: "ca93febf097541379e21c45ca870266f" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 38163 } }
I20250903 08:09:16.745173 24050 catalog_manager.cc:5582] T adcd006723944e0eb7983a66b1e8e70d P ca93febf097541379e21c45ca870266f reported cstate change: term changed from 0 to 1, leader changed from <none> to ca93febf097541379e21c45ca870266f (127.12.201.193). New cstate: current_term: 1 leader_uuid: "ca93febf097541379e21c45ca870266f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c296b676ee9444394e0b09933dd3545" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 43221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c58657070824b90b752c6cd04a89eb1" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 37993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca93febf097541379e21c45ca870266f" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 38163 } health_report { overall_health: HEALTHY } } }
I20250903 08:09:16.762908 24533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:09:16.759959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51982 (local address 127.12.201.193:38163)
0903 08:09:16.760083 (+ 124us) server_negotiation.cc:207] Beginning negotiation
0903 08:09:16.760087 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:09:16.760100 (+ 13us) server_negotiation.cc:408] Connection header received
0903 08:09:16.760180 (+ 80us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:09:16.760182 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:09:16.760236 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:09:16.760315 (+ 79us) server_negotiation.cc:227] Negotiated authn=TOKEN
0903 08:09:16.760611 (+ 296us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:16.761089 (+ 478us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:09:16.761706 (+ 617us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:09:16.761923 (+ 217us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:09:16.762027 (+ 104us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0903 08:09:16.762501 (+ 474us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0903 08:09:16.762622 (+ 121us) server_negotiation.cc:1036] Waiting for connection context
0903 08:09:16.762706 (+ 84us) server_negotiation.cc:300] Negotiation successful
0903 08:09:16.762764 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20250903 08:09:16.768383 24335 raft_consensus.cc:1273] T adcd006723944e0eb7983a66b1e8e70d P 1c58657070824b90b752c6cd04a89eb1 [term 1 FOLLOWER]: Refusing update from remote peer ca93febf097541379e21c45ca870266f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 08:09:16.768389 24471 raft_consensus.cc:1273] T adcd006723944e0eb7983a66b1e8e70d P 3c296b676ee9444394e0b09933dd3545 [term 1 FOLLOWER]: Refusing update from remote peer ca93febf097541379e21c45ca870266f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 08:09:16.769006 24540 consensus_queue.cc:1035] T adcd006723944e0eb7983a66b1e8e70d P ca93febf097541379e21c45ca870266f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c296b676ee9444394e0b09933dd3545" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 43221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 08:09:16.769201 24545 consensus_queue.cc:1035] T adcd006723944e0eb7983a66b1e8e70d P ca93febf097541379e21c45ca870266f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c58657070824b90b752c6cd04a89eb1" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 37993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 08:09:16.778661 24550 mvcc.cc:204] Tried to move back new op lower bound from 7196208974916259840 to 7196208974818549760. Current Snapshot: MvccSnapshot[applied={T|T < 7196208974916259840}]
I20250903 08:09:16.784533 24050 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48082:
table { table_name: "test-table" } modify_external_catalogs: true
I20250903 08:09:16.784691 24050 catalog_manager.cc:2730] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48082:
table { table_name: "test-table" } modify_external_catalogs: true
I20250903 08:09:16.787230 24050 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e2f3b5eb3fc34530b7e91cf02d665309: Sending DeleteTablet for 3 replicas of tablet adcd006723944e0eb7983a66b1e8e70d
I20250903 08:09:16.788086 24179 tablet_service.cc:1515] Processing DeleteTablet for tablet adcd006723944e0eb7983a66b1e8e70d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 08:09:16 UTC) from {username='kudu', principal='kudu/127.12.201.254@KRBTEST.COM'} at 127.0.0.1:51974
I20250903 08:09:16.788100 24451 tablet_service.cc:1515] Processing DeleteTablet for tablet adcd006723944e0eb7983a66b1e8e70d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 08:09:16 UTC) from {username='kudu', principal='kudu/127.12.201.254@KRBTEST.COM'} at 127.0.0.1:54184
I20250903 08:09:16.788244 24315 tablet_service.cc:1515] Processing DeleteTablet for tablet adcd006723944e0eb7983a66b1e8e70d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 08:09:16 UTC) from {username='kudu', principal='kudu/127.12.201.254@KRBTEST.COM'} at 127.0.0.1:37638
I20250903 08:09:16.788806 24560 tablet_replica.cc:331] T adcd006723944e0eb7983a66b1e8e70d P 1c58657070824b90b752c6cd04a89eb1: stopping tablet replica
I20250903 08:09:16.788864 24558 tablet_replica.cc:331] T adcd006723944e0eb7983a66b1e8e70d P 3c296b676ee9444394e0b09933dd3545: stopping tablet replica
I20250903 08:09:16.789146 24560 raft_consensus.cc:2241] T adcd006723944e0eb7983a66b1e8e70d P 1c58657070824b90b752c6cd04a89eb1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 08:09:16.789194 24558 raft_consensus.cc:2241] T adcd006723944e0eb7983a66b1e8e70d P 3c296b676ee9444394e0b09933dd3545 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 08:09:16.789227 13095 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUML90r/build/debug/bin/kudu with pid 24116
I20250903 08:09:16.789559 24560 raft_consensus.cc:2270] T adcd006723944e0eb7983a66b1e8e70d P 1c58657070824b90b752c6cd04a89eb1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 08:09:16.790402 24558 raft_consensus.cc:2270] T adcd006723944e0eb7983a66b1e8e70d P 3c296b676ee9444394e0b09933dd3545 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 08:09:16.791342 24560 ts_tablet_manager.cc:1905] T adcd006723944e0eb7983a66b1e8e70d P 1c58657070824b90b752c6cd04a89eb1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 08:09:16.792016 24558 ts_tablet_manager.cc:1905] T adcd006723944e0eb7983a66b1e8e70d P 3c296b676ee9444394e0b09933dd3545: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 08:09:16.794030 24558 ts_tablet_manager.cc:1918] T adcd006723944e0eb7983a66b1e8e70d P 3c296b676ee9444394e0b09933dd3545: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250903 08:09:16.794121 24558 log.cc:1199] T adcd006723944e0eb7983a66b1e8e70d P 3c296b676ee9444394e0b09933dd3545: Deleting WAL directory at /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/minicluster-data/ts-2/wal/wals/adcd006723944e0eb7983a66b1e8e70d
I20250903 08:09:16.794459 24558 ts_tablet_manager.cc:1939] T adcd006723944e0eb7983a66b1e8e70d P 3c296b676ee9444394e0b09933dd3545: Deleting consensus metadata
I20250903 08:09:16.794646 24560 ts_tablet_manager.cc:1918] T adcd006723944e0eb7983a66b1e8e70d P 1c58657070824b90b752c6cd04a89eb1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250903 08:09:16.794747 24560 log.cc:1199] T adcd006723944e0eb7983a66b1e8e70d P 1c58657070824b90b752c6cd04a89eb1: Deleting WAL directory at /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756886860253814-13095-0/minicluster-data/ts-1/wal/wals/adcd006723944e0eb7983a66b1e8e70d
I20250903 08:09:16.795171 24560 ts_tablet_manager.cc:1939] T adcd006723944e0eb7983a66b1e8e70d P 1c58657070824b90b752c6cd04a89eb1: Deleting consensus metadata
I20250903 08:09:16.795729 24036 catalog_manager.cc:4928] TS 3c296b676ee9444394e0b09933dd3545 (127.12.201.195:43221): tablet adcd006723944e0eb7983a66b1e8e70d (table test-table [id=9b0a57e5a93649319dccbd42e6338931]) successfully deleted
I20250903 08:09:16.796283 24036 catalog_manager.cc:4928] TS 1c58657070824b90b752c6cd04a89eb1 (127.12.201.194:37993): tablet adcd006723944e0eb7983a66b1e8e70d (table test-table [id=9b0a57e5a93649319dccbd42e6338931]) successfully deleted
W20250903 08:09:16.797127 24039 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.201.193:38163 (error 108)
I20250903 08:09:16.797886 13095 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUML90r/build/debug/bin/kudu with pid 24252
I20250903 08:09:16.798233 24530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:09:16.797884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.193:38163 (local address 127.0.0.1:51984)
0903 08:09:16.798062 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0903 08:09:16.798132 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.201.193:38163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20250903 08:09:16.800935 24039 catalog_manager.cc:4671] TS ca93febf097541379e21c45ca870266f (127.12.201.193:38163): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet adcd006723944e0eb7983a66b1e8e70d: Network error: Client connection negotiation failed: client connection to 127.12.201.193:38163: connect: Connection refused (error 111)
I20250903 08:09:16.803790 13095 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUML90r/build/debug/bin/kudu with pid 24388
I20250903 08:09:16.809496 13095 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUML90r/build/debug/bin/kudu with pid 24020
2025-09-03T08:09:16Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (3242 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 03 08:09:16 dist-test-slave-27r9 krb5kdc[24564](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 03 08:09:16 dist-test-slave-27r9 krb5kdc[24564](info): set up 2 sockets
Sep 03 08:09:16 dist-test-slave-27r9 krb5kdc[24564](info): commencing operation
krb5kdc: starting...
W20250903 08:09:18.807821 13095 mini_kdc.cc:121] Time spent starting KDC: real 1.976s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/krb5kdc/test-user.keytab.
Sep 03 08:09:18 dist-test-slave-27r9 krb5kdc[24564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-09-03T08:09:18Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-03T08:09:18Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.12.201.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.12.201.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.12.201.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/krb5kdc/rangeradmin_127.12.201.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.12.201.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/krb5kdc/rangeradmin_127.12.201.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.12.201.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.12.201.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.12.201.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/krb5kdc/rangerlookup_127.12.201.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.12.201.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/krb5kdc/rangerlookup_127.12.201.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.12.201.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.201.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.201.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/krb5kdc/HTTP_127.12.201.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.12.201.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/krb5kdc/HTTP_127.12.201.212@KRBTEST.COM.keytab.
I20250903 08:09:18.973562 13095 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default "max_connections" ... 100
selecting default "shared_buffers" ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskUML90r/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/minicluster-data/postgres -l logfile start
2025-09-03 08:09:21.551 UTC [24595] LOG: starting PostgreSQL 17.2 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
2025-09-03 08:09:21.551 UTC [24595] LOG: listening on IPv4 address "127.12.201.212", port 41195
2025-09-03 08:09:21.555 UTC [24595] LOG: listening on Unix socket "/tmp/.s.PGSQL.41195"
2025-09-03 08:09:21.561 UTC [24600] LOG: database system was shut down at 2025-09-03 08:09:20 UTC
2025-09-03 08:09:21.565 UTC [24595] LOG: database system is ready to accept connections
I20250903 08:09:23.459661 13095 mini_postgres.cc:96] Postgres bound to 41195
2025-09-03 08:09:23.465 UTC [24606] FATAL: database "slave" does not exist
127.12.201.212:41195 - accepting connections
I20250903 08:09:23.466316 13095 mini_ranger.cc:161] Starting Ranger...
I20250903 08:09:23.483922 13095 mini_ranger.cc:84] Created miniranger Postgres user
I20250903 08:09:23.552933 13095 mini_ranger.cc:87] Created ranger Postgres database
I20250903 08:09:23.553025 13095 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/minicluster-data/ranger-admin
2025-09-03 08:09:23,859 [I] DB FLAVOR :POSTGRES
2025-09-03 08:09:23,859 [I] --------- Verifying Ranger DB connection ---------
2025-09-03 08:09:23,859 [I] Checking connection..
2025-09-03 08:09:23,859 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUML90r/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUML90r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.201.212:41195/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2025-09-03 08:09:24,193 [I] Checking connection passed.
2025-09-03 08:09:24,193 [I] --------- Verifying version history table ---------
2025-09-03 08:09:24,194 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUML90r/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUML90r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.201.212:41195/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2025-09-03 08:09:24,499 [I] Table x_db_version_h does not exist in database ranger
2025-09-03 08:09:24,499 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUML90r/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUML90r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.201.212:41195/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2025-09-03 08:09:24,823 [I] Table x_db_version_h does not exist in database ranger
2025-09-03 08:09:24,824 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2025-09-03 08:09:24,824 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUML90r/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUML90r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.201.212:41195/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskUML90r/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2025-09-03 08:09:25.131 UTC [24673] WARNING: there is no transaction in progress
2025-09-03 08:09:25,137 [I] create_dbversion_catalog.sql file imported successfully
2025-09-03 08:09:25,138 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUML90r/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUML90r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.201.212:41195/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2025-09-03 08:09:25,475 [I] Table x_db_version_h already exists in database 'ranger'
2025-09-03 08:09:25,475 [I] --------- Importing Ranger Core DB Schema ---------
2025-09-03 08:09:25,475 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUML90r/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUML90r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.201.212:41195/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2025-09-03 08:09:25,782 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUML90r/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUML90r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.201.212:41195/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2025-09-03 08:09:26,095 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUML90r/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUML90r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.201.212:41195/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-27r9.c.gcp-upstream.internal','N') ;"
2025-09-03 08:09:26,412 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUML90r/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUML90r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.201.212:41195/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2025-09-03 08:09:26,729 [I] Table x_portal_user does not exist in database ranger
2025-09-03 08:09:26,729 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUML90r/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUML90r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.201.212:41195/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2025-09-03 08:09:27,032 [I] Table x_policy_ref_group does not exist in database ranger
2025-09-03 08:09:27,033 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUML90r/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUML90r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.201.212:41195/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2025-09-03 08:09:27,361 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUML90r/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUML90r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.201.212:41195/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2025-09-03 08:09:27,678 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2025-09-03 08:09:27,678 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUML90r/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUML90r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.201.212:41195/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskUML90r/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2025-09-03 08:09:28.490 UTC [24790] WARNING: there is no transaction in progress
2025-09-03 08:09:28.505 UTC [24790] WARNING: there is no transaction in progress
2025-09-03 08:09:28.520 UTC [24790] WARNING: there is no transaction in progress
2025-09-03 08:09:28.534 UTC [24790] WARNING: there is no transaction in progress
2025-09-03 08:09:28.549 UTC [24790] WARNING: there is no transaction in progress
2025-09-03 08:09:28.563 UTC [24790] WARNING: there is no transaction in progress
2025-09-03 08:09:28.610 UTC [24790] WARNING: there is no transaction in progress
2025-09-03 08:09:28.615 UTC [24790] WARNING: there is no transaction in progress
2025-09-03 08:09:28.624 UTC [24790] WARNING: there is no transaction in progress
2025-09-03 08:09:28.634 UTC [24790] WARNING: there is no transaction in progress
2025-09-03 08:09:28.643 UTC [24790] WARNING: there is no transaction in progress
2025-09-03 08:09:28.653 UTC [24790] WARNING: there is no transaction in progress
2025-09-03 08:09:28.661 UTC [24790] WARNING: there is no transaction in progress
2025-09-03 08:09:28.665 UTC [24790] WARNING: there is no transaction in progress
2025-09-03 08:09:28.668 UTC [24790] WARNING: there is no transaction in progress
2025-09-03 08:09:28.672 UTC [24790] WARNING: there is no transaction in progress
2025-09-03 08:09:29,079 [I] ranger_core_db_postgres.sql file imported successfully
2025-09-03 08:09:29,080 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUML90r/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUML90r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.201.212:41195/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2025-09-03 08:09:29,403 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2025-09-03 08:09:29,403 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUML90r/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUML90r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.201.212:41195/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2025-09-03 08:09:29,735 [I] Table x_portal_user already exists in database 'ranger'
2025-09-03 08:09:29,735 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUML90r/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUML90r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.201.212:41195/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2025-09-03 08:09:30,049 [I] Table x_policy_ref_group already exists in database 'ranger'
2025-09-03 08:09:30,049 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUML90r/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUML90r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.201.212:41195/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2025-09-03 08:09:30,368 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUML90r/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUML90r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.201.212:41195/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2025-09-03 08:09:30,675 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUML90r/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUML90r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.201.212:41195/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-27r9.c.gcp-upstream.internal';"
2025-09-03 08:09:30,991 [I] CORE_DB_SCHEMA import status has been updated
2025-09-03 08:09:30,992 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUML90r/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUML90r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.201.212:41195/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2025-09-03 08:09:31,322 [I] DB_PATCHES have already been applied
I20250903 08:09:31.328508 13095 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/minicluster-data/ranger-admin:/tmp/dist-test-taskUML90r/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskUML90r/thirdparty/src/hadoop-3.4.1/*:/tmp/dist-test-taskUML90r/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUML90r/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20250903 08:09:31.328575 13095 mini_ranger.cc:193] Using host: 127.12.201.212
I20250903 08:09:31.331622 13095 mini_ranger.cc:227] Ranger admin URL: http://127.12.201.212:33175
Sep 03, 2025 8:09:31 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 03, 2025 8:09:31 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 03, 2025 8:09:31 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/minicluster-data/ranger-admin/ews/webapp
Sep 03, 2025 8:09:31 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 03, 2025 8:09:31 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 03, 2025 8:09:31 AM org.apache.catalina.core.StandardContext setPath
WARNING: A context path must either be an empty string or start with a '/' and do not end with a '/'. The path [/] does not meet these criteria and has been changed to []
Sep 03, 2025 8:09:32 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/minicluster-data/ranger-admin/ews/webapp].
Sep 03, 2025 8:09:32 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
Sep 03, 2025 8:09:32 AM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-bio-33175"]
Sep 03, 2025 8:09:32 AM org.apache.catalina.core.StandardService startInternal
INFO: Starting service [Tomcat]
Sep 03, 2025 8:09:32 AM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet Engine: Apache Tomcat/7.0.105
Sep 03, 2025 8:09:32 AM org.apache.catalina.loader.WebappClassLoaderBase validateJarFile
INFO: validateJarFile(/tmp/dist-test-taskUML90r/thirdparty/src/ranger-2.1.0-admin/ews/webapp/WEB-INF/lib/javax.servlet-api-3.1.0.jar) - jar not loaded. See Servlet Spec 3.0, section 10.7.2. Offending class: javax/servlet/Servlet.class
Sep 03, 2025 8:09:32 AM org.apache.catalina.loader.WebappClassLoaderBase validateJarFile
INFO: validateJarFile(/tmp/dist-test-taskUML90r/thirdparty/src/ranger-2.1.0-admin/ews/webapp/WEB-INF/lib/tomcat-el-api-7.0.105.jar) - jar not loaded. See Servlet Spec 3.0, section 10.7.2. Offending class: javax/el/Expression.class
Sep 03, 2025 8:09:32 AM org.apache.catalina.startup.ContextConfig getDefaultWebXmlFragment
INFO: No global web.xml found
Sep 03, 2025 8:09:33 AM org.apache.catalina.startup.TaglibUriRule body
INFO: TLD skipped. URI: urn:com:sun:jersey:api:view is already defined
Sep 03, 2025 8:09:33 AM org.apache.catalina.startup.TldConfig execute
INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
Sep 03, 2025 8:09:33 AM org.apache.catalina.core.ApplicationContext log
INFO: Set web app root system property: 'webapp.root' = [/tmp/dist-test-taskUML90r/thirdparty/src/ranger-2.1.0-admin/ews/webapp/]
Sep 03, 2025 8:09:33 AM org.apache.catalina.core.ApplicationContext log
INFO: Initializing Spring root WebApplicationContext
I20250903 08:09:33.574733 13095 mini_ranger.cc:160] Time spent starting Ranger: real 10.108s user 0.001s sys 0.009s
Sep 03, 2025 8:09:52 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 03, 2025 8:09:52 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.AssetREST
Sep 03, 2025 8:09:52 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 03, 2025 8:09:52 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 03, 2025 8:09:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 03, 2025 8:09:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 03, 2025 8:09:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 03, 2025 8:09:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 03, 2025 8:09:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 03, 2025 8:09:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 03, 2025 8:09:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 03, 2025 8:09:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 03, 2025 8:09:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 03, 2025 8:09:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 03, 2025 8:09:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 03, 2025 8:09:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 03, 2025 8:09:52 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 03, 2025 8:09:53 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
Sep 03, 2025 8:09:53 AM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["http-bio-33175"]
I20250903 08:09:54.099184 13095 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.12.201.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.12.201.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.12.201.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/krb5kdc/rangerkms_127.12.201.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.12.201.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/krb5kdc/rangerkms_127.12.201.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.12.201.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.12.201.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.201.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/krb5kdc/HTTP_127.12.201.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.12.201.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/krb5kdc/HTTP_127.12.201.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 03 08:09:54 dist-test-slave-27r9 krb5kdc[24564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886994, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20250903 08:09:54.178256 13095 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20250903 08:09:54.187196 13095 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20250903 08:09:54.295949 13095 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20250903 08:09:54.296061 13095 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/minicluster-data/ranger-kms
I20250903 08:09:54.296104 13095 mini_ranger_kms.cc:227] Using postgres at 127.12.201.212:41195
2025-09-03 08:09:54,480 [I] DB FLAVOR :POSTGRES
2025-09-03 08:09:54,480 [I] --------- Verifying Ranger DB connection ---------
2025-09-03 08:09:54,481 [I] Checking connection
2025-09-03 08:09:54,481 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUML90r/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.201.212:41195/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2025-09-03 08:09:54,810 [I] connection success
2025-09-03 08:09:54,810 [I] --------- Verifying Ranger DB tables ---------
2025-09-03 08:09:54,811 [I] Verifying table ranger_masterkey in database rangerkms
2025-09-03 08:09:54,811 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUML90r/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.201.212:41195/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2025-09-03 08:09:55,117 [I] Table ranger_masterkey does not exist in database rangerkms
2025-09-03 08:09:55,117 [I] --------- Importing Ranger Core DB Schema ---------
2025-09-03 08:09:55,117 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2025-09-03 08:09:55,117 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUML90r/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.201.212:41195/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2025-09-03 08:09:55,450 [I] kms_core_db_postgres.sql DB schema imported successfully
I20250903 08:09:55.934175 13095 mini_ranger_kms.cc:310] Created kms service
I20250903 08:09:56.128626 13095 mini_ranger_kms.cc:326] Created kudu user
I20250903 08:09:56.170447 13095 mini_ranger_kms.cc:343] Created rangerkms user
Sep 03 08:09:56 dist-test-slave-27r9 krb5kdc[24564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886994, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.12.201.212@KRBTEST.COM
I20250903 08:09:56.539793 13095 mini_ranger_kms.cc:384] Added ranger policy
I20250903 08:09:56.539964 13095 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/minicluster-data/ranger-kms:/tmp/dist-test-taskUML90r/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUML90r/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskUML90r/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20250903 08:09:56.539984 13095 mini_ranger_kms.cc:240] Using host: 127.12.201.212
I20250903 08:09:56.543195 13095 mini_ranger_kms.cc:276] Ranger KMS PID: 24982
I20250903 08:09:56.543311 13095 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.12.201.212:47559
Sep 03, 2025 8:09:57 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 03, 2025 8:09:57 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 03, 2025 8:09:57 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 03, 2025 8:09:57 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 03, 2025 8:09:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/minicluster-data/ranger-kms/ews/webapp].
Sep 03, 2025 8:09:58 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 03, 2025 8:09:58 AM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-bio-47559"]
Sep 03, 2025 8:09:58 AM org.apache.catalina.core.StandardService startInternal
INFO: Starting service [Tomcat]
Sep 03, 2025 8:09:58 AM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet Engine: Apache Tomcat/7.0.105
Sep 03, 2025 8:09:58 AM org.apache.catalina.startup.ContextConfig getDefaultWebXmlFragment
INFO: No global web.xml found
I20250903 08:09:58.713899 13095 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.536s user 0.003s sys 0.163s
I20250903 08:09:58.714035 13095 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20250903 08:09:58.714120 13095 mini_ranger_kms.cc:401] 127.12.201.212:47559/kms/v1/keys
Sep 03, 2025 8:09:58 AM org.apache.catalina.startup.TldConfig execute
INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
Sep 03 08:09:58 dist-test-slave-27r9 krb5kdc[24564](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1756886998, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.201.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 03 08:09:59 dist-test-slave-27r9 krb5kdc[24564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886998, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.201.212@KRBTEST.COM for HTTP/127.12.201.212@KRBTEST.COM
Sep 03 08:09:59 dist-test-slave-27r9 krb5kdc[24564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756886998, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.201.212@KRBTEST.COM for HTTP/127.12.201.212@KRBTEST.COM
Sep 03 08:10:02 dist-test-slave-27r9 krb5kdc[24564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756887002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.12.201.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.201.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.201.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.201.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.201.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.201.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.201.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.201.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/krb5kdc/kudu.keytab.
I20250903 08:10:02.157893 13095 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUML90r/build/debug/bin/kudu
/tmp/dist-test-taskUML90r/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.201.254:44561
--webserver_interface=127.12.201.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.201.212:39951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.201.254:44561
--ranger_config_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.201.212:47559/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 08:10:02.265023 25035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:10:02.265239 25035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:10:02.265292 25035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:10:02.268471 25035 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250903 08:10:02.268538 25035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 08:10:02.268561 25035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:10:02.268579 25035 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250903 08:10:02.268596 25035 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250903 08:10:02.273075 25035 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.201.212:39951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/minicluster-data/master-0/wal
--ranger_kms_url=127.12.201.212:47559/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.12.201.254:44561
--ranger_config_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.201.254:44561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.201.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25035
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 07:50:08 UTC on 5fd53c4cbb9d
build id 7874
I20250903 08:10:02.274183 25035 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:10:02.275116 25035 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:10:02.281579 25041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:10:02.281661 25035 server_base.cc:1047] running on GCE node
W20250903 08:10:02.281602 25040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:10:02.281877 25043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:10:02.282444 25035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:10:02.283397 25035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:10:02.284556 25035 hybrid_clock.cc:648] HybridClock initialized: now 1756887002284528 us; error 45 us; skew 500 ppm
Sep 03 08:10:02 dist-test-slave-27r9 krb5kdc[24564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756887002, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 08:10:02.287310 25035 init.cc:377] Logged in from keytab as kudu/127.12.201.254@KRBTEST.COM (short username kudu)
I20250903 08:10:02.288578 25035 webserver.cc:480] Webserver started at http://127.12.201.254:42843/ using document root <none> and password file <none>
I20250903 08:10:02.289242 25035 fs_manager.cc:362] Metadata directory not provided
I20250903 08:10:02.289312 25035 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:10:02.289535 25035 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
Sep 03 08:10:02 dist-test-slave-27r9 krb5kdc[24564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756887002, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.254@KRBTEST.COM for HTTP/127.12.201.212@KRBTEST.COM
I20250903 08:10:02.339385 25035 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/minicluster-data/master-0/data/instance:
uuid: "61b76a5beaa947ea8703e0e50b0e3de4"
format_stamp: "Formatted at 2025-09-03 08:10:02 on dist-test-slave-27r9"
server_key: "6e0651e833f0ed888b8e053714854b26"
server_key_iv: "1d06b12c49829ab24ee28d9d23834f23"
server_key_version: "kuduclusterkey@0"
I20250903 08:10:02.340011 25035 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/minicluster-data/master-0/wal/instance:
uuid: "61b76a5beaa947ea8703e0e50b0e3de4"
format_stamp: "Formatted at 2025-09-03 08:10:02 on dist-test-slave-27r9"
server_key: "6e0651e833f0ed888b8e053714854b26"
server_key_iv: "1d06b12c49829ab24ee28d9d23834f23"
server_key_version: "kuduclusterkey@0"
I20250903 08:10:02.344522 25035 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20250903 08:10:02.358333 25050 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:10:02.359773 25035 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250903 08:10:02.359928 25035 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/minicluster-data/master-0/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/minicluster-data/master-0/wal
uuid: "61b76a5beaa947ea8703e0e50b0e3de4"
format_stamp: "Formatted at 2025-09-03 08:10:02 on dist-test-slave-27r9"
server_key: "6e0651e833f0ed888b8e053714854b26"
server_key_iv: "1d06b12c49829ab24ee28d9d23834f23"
server_key_version: "kuduclusterkey@0"
I20250903 08:10:02.360047 25035 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:10:02.378710 25035 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:10:02.381621 25035 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:10:02.381788 25035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:10:02.390229 25035 rpc_server.cc:307] RPC server started. Bound to: 127.12.201.254:44561
I20250903 08:10:02.390308 25102 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.201.254:44561 every 8 connection(s)
I20250903 08:10:02.391286 25035 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/minicluster-data/master-0/data/info.pb
I20250903 08:10:02.393783 25103 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:10:02.394372 13095 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUML90r/build/debug/bin/kudu as pid 25035
I20250903 08:10:02.394506 13095 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/minicluster-data/master-0/wal/instance
I20250903 08:10:02.394780 13095 external_mini_cluster.cc:1468] Setting key 442c7bc219dac7a2a1a42f1d3eaf610c
I20250903 08:10:02.400899 25103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61b76a5beaa947ea8703e0e50b0e3de4: Bootstrap starting.
I20250903 08:10:02.403383 25103 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 61b76a5beaa947ea8703e0e50b0e3de4: Neither blocks nor log segments found. Creating new log.
Sep 03 08:10:02 dist-test-slave-27r9 krb5kdc[24564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756887002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.201.254@KRBTEST.COM
I20250903 08:10:02.404229 25103 log.cc:826] T 00000000000000000000000000000000 P 61b76a5beaa947ea8703e0e50b0e3de4: Log is configured to *not* fsync() on all Append() calls
I20250903 08:10:02.406162 25103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61b76a5beaa947ea8703e0e50b0e3de4: No bootstrap required, opened a new log
I20250903 08:10:02.409374 25103 raft_consensus.cc:357] T 00000000000000000000000000000000 P 61b76a5beaa947ea8703e0e50b0e3de4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61b76a5beaa947ea8703e0e50b0e3de4" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 44561 } }
I20250903 08:10:02.409621 25103 raft_consensus.cc:383] T 00000000000000000000000000000000 P 61b76a5beaa947ea8703e0e50b0e3de4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:10:02.409691 25103 raft_consensus.cc:738] T 00000000000000000000000000000000 P 61b76a5beaa947ea8703e0e50b0e3de4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b76a5beaa947ea8703e0e50b0e3de4, State: Initialized, Role: FOLLOWER
I20250903 08:10:02.410211 25103 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61b76a5beaa947ea8703e0e50b0e3de4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61b76a5beaa947ea8703e0e50b0e3de4" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 44561 } }
I20250903 08:10:02.410404 25103 raft_consensus.cc:397] T 00000000000000000000000000000000 P 61b76a5beaa947ea8703e0e50b0e3de4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250903 08:10:02.410475 25103 raft_consensus.cc:491] T 00000000000000000000000000000000 P 61b76a5beaa947ea8703e0e50b0e3de4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250903 08:10:02.410606 25103 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 61b76a5beaa947ea8703e0e50b0e3de4 [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:10:02.411078 25106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:10:02.396107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38500 (local address 127.12.201.254:44561)
0903 08:10:02.396523 (+ 416us) server_negotiation.cc:207] Beginning negotiation
0903 08:10:02.396538 (+ 15us) server_negotiation.cc:400] Waiting for connection header
0903 08:10:02.396582 (+ 44us) server_negotiation.cc:408] Connection header received
0903 08:10:02.397457 (+ 875us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:10:02.397483 (+ 26us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:10:02.397893 (+ 410us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:10:02.398253 (+ 360us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:10:02.399355 (+ 1102us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:10:02.400485 (+ 1130us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:10:02.401394 (+ 909us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:10:02.401646 (+ 252us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:10:02.404508 (+ 2862us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:10:02.404529 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:10:02.404544 (+ 15us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:10:02.404589 (+ 45us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:10:02.408048 (+ 3459us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:10:02.408603 (+ 555us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:10:02.408609 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:10:02.408616 (+ 7us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:10:02.408699 (+ 83us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:10:02.408998 (+ 299us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:10:02.409001 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:10:02.409003 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:10:02.409441 (+ 438us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:10:02.409643 (+ 202us) server_negotiation.cc:1036] Waiting for connection context
0903 08:10:02.410024 (+ 381us) server_negotiation.cc:300] Negotiation successful
0903 08:10:02.410261 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":114,"threads_started":1}
I20250903 08:10:02.411599 25103 raft_consensus.cc:513] T 00000000000000000000000000000000 P 61b76a5beaa947ea8703e0e50b0e3de4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61b76a5beaa947ea8703e0e50b0e3de4" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 44561 } }
I20250903 08:10:02.411937 25103 leader_election.cc:304] T 00000000000000000000000000000000 P 61b76a5beaa947ea8703e0e50b0e3de4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 61b76a5beaa947ea8703e0e50b0e3de4; no voters:
I20250903 08:10:02.412236 25103 leader_election.cc:290] T 00000000000000000000000000000000 P 61b76a5beaa947ea8703e0e50b0e3de4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250903 08:10:02.412366 25108 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 61b76a5beaa947ea8703e0e50b0e3de4 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 08:10:02.412559 25108 raft_consensus.cc:695] T 00000000000000000000000000000000 P 61b76a5beaa947ea8703e0e50b0e3de4 [term 1 LEADER]: Becoming Leader. State: Replica: 61b76a5beaa947ea8703e0e50b0e3de4, State: Running, Role: LEADER
I20250903 08:10:02.412889 25108 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61b76a5beaa947ea8703e0e50b0e3de4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61b76a5beaa947ea8703e0e50b0e3de4" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 44561 } }
I20250903 08:10:02.413443 25103 sys_catalog.cc:564] T 00000000000000000000000000000000 P 61b76a5beaa947ea8703e0e50b0e3de4 [sys.catalog]: configured and running, proceeding with master startup.
I20250903 08:10:02.414382 25108 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61b76a5beaa947ea8703e0e50b0e3de4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "61b76a5beaa947ea8703e0e50b0e3de4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61b76a5beaa947ea8703e0e50b0e3de4" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 44561 } } }
I20250903 08:10:02.414584 25108 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61b76a5beaa947ea8703e0e50b0e3de4 [sys.catalog]: This master's current role is: LEADER
I20250903 08:10:02.414832 25108 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61b76a5beaa947ea8703e0e50b0e3de4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61b76a5beaa947ea8703e0e50b0e3de4. Latest consensus state: current_term: 1 leader_uuid: "61b76a5beaa947ea8703e0e50b0e3de4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61b76a5beaa947ea8703e0e50b0e3de4" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 44561 } } }
I20250903 08:10:02.414960 25108 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61b76a5beaa947ea8703e0e50b0e3de4 [sys.catalog]: This master's current role is: LEADER
I20250903 08:10:02.415323 25117 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250903 08:10:02.419106 25117 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250903 08:10:02.419534 25103 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20250903 08:10:02.430770 25117 catalog_manager.cc:1349] Generated new cluster ID: 4a4ded0571074d64abb6e265e10e6bba
I20250903 08:10:02.430938 25117 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250903 08:10:02.440562 25117 catalog_manager.cc:1372] Generated new certificate authority record
I20250903 08:10:02.441933 25117 catalog_manager.cc:1506] Loading token signing keys...
I20250903 08:10:02.451725 25117 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 61b76a5beaa947ea8703e0e50b0e3de4: Generated new TSK 0
I20250903 08:10:02.452682 25117 catalog_manager.cc:1516] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.201.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.201.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.201.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.201.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.201.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.201.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.201.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.201.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/krb5kdc/kudu.keytab.
I20250903 08:10:03.222321 13095 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUML90r/build/debug/bin/kudu
/tmp/dist-test-taskUML90r/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.201.212:47559/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.201.193:0
--local_ip_for_outbound_sockets=127.12.201.193
--webserver_interface=127.12.201.193
--webserver_port=0
--tserver_master_addrs=127.12.201.254:44561
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.201.212:39951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 08:10:03.345669 25149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:10:03.345901 25149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:10:03.345959 25149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:10:03.349324 25149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 08:10:03.349392 25149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:10:03.349493 25149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.201.193
I20250903 08:10:03.354179 25149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.201.212:39951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.12.201.212:47559/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.201.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.201.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.12.201.254:44561
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.201.193
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 07:50:08 UTC on 5fd53c4cbb9d
build id 7874
I20250903 08:10:03.355571 25149 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:10:03.356638 25149 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:10:03.364434 25156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:10:03.364768 25159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:10:03.364988 25157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:10:03.365439 25149 server_base.cc:1047] running on GCE node
I20250903 08:10:03.365888 25149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:10:03.366534 25149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:10:03.367695 25149 hybrid_clock.cc:648] HybridClock initialized: now 1756887003367681 us; error 34 us; skew 500 ppm
Sep 03 08:10:03 dist-test-slave-27r9 krb5kdc[24564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756887003, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 08:10:03.371022 25149 init.cc:377] Logged in from keytab as kudu/127.12.201.193@KRBTEST.COM (short username kudu)
I20250903 08:10:03.372212 25149 webserver.cc:480] Webserver started at http://127.12.201.193:45653/ using document root <none> and password file <none>
I20250903 08:10:03.372820 25149 fs_manager.cc:362] Metadata directory not provided
I20250903 08:10:03.372876 25149 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:10:03.373066 25149 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
Sep 03 08:10:03 dist-test-slave-27r9 krb5kdc[24564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756887003, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.193@KRBTEST.COM for HTTP/127.12.201.212@KRBTEST.COM
I20250903 08:10:03.401988 25149 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/minicluster-data/ts-0/data/instance:
uuid: "6492ed940b264ca4bccc57bf3b00e4ea"
format_stamp: "Formatted at 2025-09-03 08:10:03 on dist-test-slave-27r9"
server_key: "d17c926d50a401415e2a8fbec2280ef2"
server_key_iv: "2bcd3ec093cb490671b97ec80360fc78"
server_key_version: "kuduclusterkey@0"
I20250903 08:10:03.402886 25149 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/minicluster-data/ts-0/wal/instance:
uuid: "6492ed940b264ca4bccc57bf3b00e4ea"
format_stamp: "Formatted at 2025-09-03 08:10:03 on dist-test-slave-27r9"
server_key: "d17c926d50a401415e2a8fbec2280ef2"
server_key_iv: "2bcd3ec093cb490671b97ec80360fc78"
server_key_version: "kuduclusterkey@0"
I20250903 08:10:03.412380 25149 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.003s sys 0.001s
I20250903 08:10:03.429440 25168 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:10:03.430752 25149 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250903 08:10:03.430884 25149 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/minicluster-data/ts-0/wal
uuid: "6492ed940b264ca4bccc57bf3b00e4ea"
format_stamp: "Formatted at 2025-09-03 08:10:03 on dist-test-slave-27r9"
server_key: "d17c926d50a401415e2a8fbec2280ef2"
server_key_iv: "2bcd3ec093cb490671b97ec80360fc78"
server_key_version: "kuduclusterkey@0"
I20250903 08:10:03.431046 25149 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:10:03.447361 25149 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:10:03.451941 25149 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:10:03.452239 25149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:10:03.453008 25149 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 08:10:03.454162 25149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 08:10:03.454268 25149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 08:10:03.454391 25149 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 08:10:03.454473 25149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 08:10:03.466007 25149 rpc_server.cc:307] RPC server started. Bound to: 127.12.201.193:44041
I20250903 08:10:03.466415 25281 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.201.193:44041 every 8 connection(s)
I20250903 08:10:03.467388 25149 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/minicluster-data/ts-0/data/info.pb
I20250903 08:10:03.469926 13095 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUML90r/build/debug/bin/kudu as pid 25149
I20250903 08:10:03.470082 13095 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/minicluster-data/ts-0/wal/instance
I20250903 08:10:03.470361 13095 external_mini_cluster.cc:1468] Setting key fb56b8477a8e2b6b7400a594e80224d8
Sep 03 08:10:03 dist-test-slave-27r9 krb5kdc[24564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756887003, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.193@KRBTEST.COM for kudu/127.12.201.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.201.194@KRBTEST.COM; defaulting to no policy
I20250903 08:10:03.483436 25284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:10:03.469496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.193:51175 (local address 127.12.201.254:44561)
0903 08:10:03.469818 (+ 322us) server_negotiation.cc:207] Beginning negotiation
0903 08:10:03.469825 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0903 08:10:03.472200 (+ 2375us) server_negotiation.cc:408] Connection header received
0903 08:10:03.472254 (+ 54us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:10:03.472260 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:10:03.472323 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:10:03.472412 (+ 89us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:10:03.473938 (+ 1526us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:10:03.474804 (+ 866us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:10:03.476179 (+ 1375us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:10:03.476395 (+ 216us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:10:03.478582 (+ 2187us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:10:03.478604 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:10:03.478608 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:10:03.478639 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:10:03.480735 (+ 2096us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:10:03.481337 (+ 602us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:10:03.481342 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:10:03.481344 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:10:03.481401 (+ 57us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:10:03.481743 (+ 342us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:10:03.481747 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:10:03.481749 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:10:03.481944 (+ 195us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:10:03.482041 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
0903 08:10:03.483088 (+ 1047us) server_negotiation.cc:300] Negotiation successful
0903 08:10:03.483254 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":95,"threads_started":1}
I20250903 08:10:03.484400 25285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:10:03.469807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:44561 (local address 127.12.201.193:51175)
0903 08:10:03.470900 (+ 1093us) negotiation.cc:107] Waiting for socket to connect
0903 08:10:03.470945 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0903 08:10:03.471907 (+ 962us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:10:03.472623 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:10:03.472635 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:10:03.473007 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:10:03.473761 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:10:03.473780 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:10:03.474961 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:10:03.474966 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:10:03.475602 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:10:03.475613 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:10:03.475845 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:10:03.476652 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:10:03.476686 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:10:03.478395 (+ 1709us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:10:03.480879 (+ 2484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:10:03.480885 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:10:03.480900 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:10:03.481218 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:10:03.481504 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:10:03.481507 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:10:03.481509 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:10:03.481632 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:10:03.482099 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:10:03.482106 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:10:03.482409 (+ 303us) client_negotiation.cc:769] Sending connection context
0903 08:10:03.482649 (+ 240us) client_negotiation.cc:240] Negotiation successful
0903 08:10:03.482944 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":886,"thread_start_us":136,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.201.194@KRBTEST.COM" created.
I20250903 08:10:03.486027 25282 heartbeater.cc:344] Connected to a master server at 127.12.201.254:44561
I20250903 08:10:03.486289 25282 heartbeater.cc:461] Registering TS with master...
I20250903 08:10:03.486953 25282 heartbeater.cc:507] Master 127.12.201.254:44561 requested a full tablet report, sending...
I20250903 08:10:03.488664 25067 ts_manager.cc:194] Registered new tserver with Master: 6492ed940b264ca4bccc57bf3b00e4ea (127.12.201.193:44041)
I20250903 08:10:03.490274 25067 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.201.193@KRBTEST.COM'} at 127.12.201.193:51175
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.201.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.201.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Sep 03 08:10:03 dist-test-slave-27r9 krb5kdc[24564](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1756887003, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
WARNING: no policy specified for HTTP/127.12.201.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.201.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.201.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.201.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/krb5kdc/kudu.keytab.
I20250903 08:10:03.535938 13095 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUML90r/build/debug/bin/kudu
/tmp/dist-test-taskUML90r/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.201.212:47559/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.201.194:0
--local_ip_for_outbound_sockets=127.12.201.194
--webserver_interface=127.12.201.194
--webserver_port=0
--tserver_master_addrs=127.12.201.254:44561
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.201.212:39951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 08:10:03.695429 25290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:10:03.695720 25290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:10:03.695822 25290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:10:03.700472 25290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 08:10:03.700562 25290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:10:03.700675 25290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.201.194
I20250903 08:10:03.706734 25290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.201.212:39951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.12.201.212:47559/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.201.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.201.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.12.201.254:44561
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25290
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.201.194
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 07:50:08 UTC on 5fd53c4cbb9d
build id 7874
I20250903 08:10:03.708215 25290 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:10:03.709378 25290 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:10:03.717291 25297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:10:03.717619 25299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:10:03.719087 25290 server_base.cc:1047] running on GCE node
W20250903 08:10:03.719484 25296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:10:03.720046 25290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:10:03.720592 25290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:10:03.721753 25290 hybrid_clock.cc:648] HybridClock initialized: now 1756887003721732 us; error 41 us; skew 500 ppm
Sep 03 08:10:03 dist-test-slave-27r9 krb5kdc[24564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756887003, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 08:10:03.724418 25290 init.cc:377] Logged in from keytab as kudu/127.12.201.194@KRBTEST.COM (short username kudu)
I20250903 08:10:03.725397 25290 webserver.cc:480] Webserver started at http://127.12.201.194:33527/ using document root <none> and password file <none>
I20250903 08:10:03.725943 25290 fs_manager.cc:362] Metadata directory not provided
I20250903 08:10:03.725996 25290 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:10:03.726188 25290 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
Sep 03 08:10:03 dist-test-slave-27r9 krb5kdc[24564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756887003, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.194@KRBTEST.COM for HTTP/127.12.201.212@KRBTEST.COM
I20250903 08:10:03.757395 25290 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/minicluster-data/ts-1/data/instance:
uuid: "39beab15348d4453bde7185836c0de2c"
format_stamp: "Formatted at 2025-09-03 08:10:03 on dist-test-slave-27r9"
server_key: "5d2056440d73ca91090124af5ad63bc3"
server_key_iv: "5dc364d2bc8296de87d6d1ed997930c3"
server_key_version: "kuduclusterkey@0"
I20250903 08:10:03.758234 25290 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/minicluster-data/ts-1/wal/instance:
uuid: "39beab15348d4453bde7185836c0de2c"
format_stamp: "Formatted at 2025-09-03 08:10:03 on dist-test-slave-27r9"
server_key: "5d2056440d73ca91090124af5ad63bc3"
server_key_iv: "5dc364d2bc8296de87d6d1ed997930c3"
server_key_version: "kuduclusterkey@0"
I20250903 08:10:03.763574 25290 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20250903 08:10:03.779737 25306 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:10:03.781136 25290 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20250903 08:10:03.781287 25290 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/minicluster-data/ts-1/wal
uuid: "39beab15348d4453bde7185836c0de2c"
format_stamp: "Formatted at 2025-09-03 08:10:03 on dist-test-slave-27r9"
server_key: "5d2056440d73ca91090124af5ad63bc3"
server_key_iv: "5dc364d2bc8296de87d6d1ed997930c3"
server_key_version: "kuduclusterkey@0"
I20250903 08:10:03.781422 25290 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:10:03.801129 25290 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:10:03.808166 25290 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:10:03.808391 25290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:10:03.809077 25290 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 08:10:03.810195 25290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 08:10:03.810266 25290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 08:10:03.810334 25290 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 08:10:03.810364 25290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 08:10:03.824646 25290 rpc_server.cc:307] RPC server started. Bound to: 127.12.201.194:43813
I20250903 08:10:03.825929 25290 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/minicluster-data/ts-1/data/info.pb
I20250903 08:10:03.827876 13095 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUML90r/build/debug/bin/kudu as pid 25290
I20250903 08:10:03.828011 13095 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/minicluster-data/ts-1/wal/instance
I20250903 08:10:03.828043 25419 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.201.194:43813 every 8 connection(s)
I20250903 08:10:03.828301 13095 external_mini_cluster.cc:1468] Setting key 770a7c6e2759e0bb232b0e8570fc11e9
Sep 03 08:10:03 dist-test-slave-27r9 krb5kdc[24564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756887003, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.194@KRBTEST.COM for kudu/127.12.201.254@KRBTEST.COM
I20250903 08:10:03.846089 25284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:10:03.831182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.194:56429 (local address 127.12.201.254:44561)
0903 08:10:03.831342 (+ 160us) server_negotiation.cc:207] Beginning negotiation
0903 08:10:03.831350 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0903 08:10:03.832121 (+ 771us) server_negotiation.cc:408] Connection header received
0903 08:10:03.833267 (+ 1146us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:10:03.833271 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:10:03.833327 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:10:03.833388 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:10:03.836203 (+ 2815us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:10:03.837010 (+ 807us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:10:03.837947 (+ 937us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:10:03.838172 (+ 225us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:10:03.841740 (+ 3568us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:10:03.841770 (+ 30us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:10:03.841774 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:10:03.841806 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:10:03.843685 (+ 1879us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:10:03.844352 (+ 667us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:10:03.844357 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:10:03.844359 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:10:03.844415 (+ 56us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:10:03.844767 (+ 352us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:10:03.844770 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:10:03.844773 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:10:03.844977 (+ 204us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:10:03.845057 (+ 80us) server_negotiation.cc:1036] Waiting for connection context
0903 08:10:03.845751 (+ 694us) server_negotiation.cc:300] Negotiation successful
0903 08:10:03.845892 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20250903 08:10:03.847851 25422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:10:03.830119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:44561 (local address 127.12.201.194:56429)
0903 08:10:03.831973 (+ 1854us) negotiation.cc:107] Waiting for socket to connect
0903 08:10:03.832021 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0903 08:10:03.833018 (+ 997us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:10:03.833821 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:10:03.833835 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:10:03.834207 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:10:03.835046 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:10:03.835064 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:10:03.837181 (+ 2117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:10:03.837184 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:10:03.837799 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:10:03.837810 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:10:03.839017 (+ 1207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:10:03.839772 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:10:03.839809 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:10:03.841528 (+ 1719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:10:03.843854 (+ 2326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:10:03.843859 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:10:03.843875 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:10:03.844228 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:10:03.844525 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:10:03.844528 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:10:03.844531 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:10:03.844660 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:10:03.845126 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:10:03.845133 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:10:03.845453 (+ 320us) client_negotiation.cc:769] Sending connection context
0903 08:10:03.846612 (+ 1159us) client_negotiation.cc:240] Negotiation successful
0903 08:10:03.846917 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1631,"thread_start_us":156,"threads_started":1}
I20250903 08:10:03.849442 25420 heartbeater.cc:344] Connected to a master server at 127.12.201.254:44561
I20250903 08:10:03.849697 25420 heartbeater.cc:461] Registering TS with master...
I20250903 08:10:03.850414 25420 heartbeater.cc:507] Master 127.12.201.254:44561 requested a full tablet report, sending...
I20250903 08:10:03.851907 25067 ts_manager.cc:194] Registered new tserver with Master: 39beab15348d4453bde7185836c0de2c (127.12.201.194:43813)
I20250903 08:10:03.852728 25067 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.201.194@KRBTEST.COM'} at 127.12.201.194:56429
WARNING: no policy specified for kudu/127.12.201.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.201.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.201.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.201.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.201.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.201.195@KRBTEST.COM" created.
Sep 03 08:10:03 dist-test-slave-27r9 krb5kdc[24564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756887003, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.254@KRBTEST.COM for HTTP/127.12.201.212@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.201.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.201.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/krb5kdc/kudu.keytab.
I20250903 08:10:03.920915 13095 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUML90r/build/debug/bin/kudu
/tmp/dist-test-taskUML90r/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.201.212:47559/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.201.195:0
--local_ip_for_outbound_sockets=127.12.201.195
--webserver_interface=127.12.201.195
--webserver_port=0
--tserver_master_addrs=127.12.201.254:44561
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.201.212:39951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 08:10:04.071033 25427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:10:04.071425 25427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:10:04.071561 25427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:10:04.076341 25427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 08:10:04.076503 25427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:10:04.076668 25427 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.201.195
I20250903 08:10:04.083117 25427 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.201.212:39951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.12.201.212:47559/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.201.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.201.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.12.201.254:44561
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25427
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.201.195
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 07:50:08 UTC on 5fd53c4cbb9d
build id 7874
I20250903 08:10:04.084623 25427 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:10:04.085741 25427 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:10:04.093724 25435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:10:04.104032 25433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:10:04.105641 25427 server_base.cc:1047] running on GCE node
W20250903 08:10:04.108788 25432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:10:04.109392 25427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:10:04.110150 25427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:10:04.119833 25427 hybrid_clock.cc:648] HybridClock initialized: now 1756887004119752 us; error 99 us; skew 500 ppm
Sep 03 08:10:04 dist-test-slave-27r9 krb5kdc[24564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756887004, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 08:10:04.123785 25427 init.cc:377] Logged in from keytab as kudu/127.12.201.195@KRBTEST.COM (short username kudu)
I20250903 08:10:04.125425 25427 webserver.cc:480] Webserver started at http://127.12.201.195:39197/ using document root <none> and password file <none>
I20250903 08:10:04.126273 25427 fs_manager.cc:362] Metadata directory not provided
I20250903 08:10:04.126411 25427 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:10:04.126693 25427 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
Sep 03 08:10:04 dist-test-slave-27r9 krb5kdc[24564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756887004, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.195@KRBTEST.COM for HTTP/127.12.201.212@KRBTEST.COM
I20250903 08:10:04.146682 25427 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/minicluster-data/ts-2/data/instance:
uuid: "d9d4d7336d5e455999d645721b3a6c0c"
format_stamp: "Formatted at 2025-09-03 08:10:04 on dist-test-slave-27r9"
server_key: "83ead74c5a7f4d31c402fd222aa3ca95"
server_key_iv: "67e3b6821086bd9c8a81ef27ba3e6033"
server_key_version: "kuduclusterkey@0"
I20250903 08:10:04.148438 25427 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/minicluster-data/ts-2/wal/instance:
uuid: "d9d4d7336d5e455999d645721b3a6c0c"
format_stamp: "Formatted at 2025-09-03 08:10:04 on dist-test-slave-27r9"
server_key: "83ead74c5a7f4d31c402fd222aa3ca95"
server_key_iv: "67e3b6821086bd9c8a81ef27ba3e6033"
server_key_version: "kuduclusterkey@0"
I20250903 08:10:04.152081 25427 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20250903 08:10:04.169997 25442 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:10:04.171484 25427 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20250903 08:10:04.171612 25427 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/minicluster-data/ts-2/wal
uuid: "d9d4d7336d5e455999d645721b3a6c0c"
format_stamp: "Formatted at 2025-09-03 08:10:04 on dist-test-slave-27r9"
server_key: "83ead74c5a7f4d31c402fd222aa3ca95"
server_key_iv: "67e3b6821086bd9c8a81ef27ba3e6033"
server_key_version: "kuduclusterkey@0"
I20250903 08:10:04.171751 25427 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:10:04.200109 25427 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:10:04.203902 25427 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:10:04.204166 25427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:10:04.204854 25427 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 08:10:04.205946 25427 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 08:10:04.206056 25427 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 08:10:04.206152 25427 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 08:10:04.206213 25427 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 08:10:04.220966 25427 rpc_server.cc:307] RPC server started. Bound to: 127.12.201.195:40721
I20250903 08:10:04.222290 25427 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/minicluster-data/ts-2/data/info.pb
I20250903 08:10:04.223325 13095 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUML90r/build/debug/bin/kudu as pid 25427
I20250903 08:10:04.223454 13095 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/minicluster-data/ts-2/wal/instance
I20250903 08:10:04.223742 13095 external_mini_cluster.cc:1468] Setting key a9c0fd667055671bee28d7080089e0bf
I20250903 08:10:04.226038 25555 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.201.195:40721 every 8 connection(s)
Sep 03 08:10:04 dist-test-slave-27r9 krb5kdc[24564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756887004, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.195@KRBTEST.COM for kudu/127.12.201.254@KRBTEST.COM
I20250903 08:10:04.243880 25284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:10:04.229148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.195:39397 (local address 127.12.201.254:44561)
0903 08:10:04.229282 (+ 134us) server_negotiation.cc:207] Beginning negotiation
0903 08:10:04.229286 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:10:04.230060 (+ 774us) server_negotiation.cc:408] Connection header received
0903 08:10:04.231246 (+ 1186us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:10:04.231251 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:10:04.231308 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:10:04.231359 (+ 51us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:10:04.233239 (+ 1880us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:10:04.234088 (+ 849us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:10:04.235047 (+ 959us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:10:04.235239 (+ 192us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:10:04.239402 (+ 4163us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:10:04.239420 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:10:04.239423 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:10:04.239455 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:10:04.241399 (+ 1944us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:10:04.242068 (+ 669us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:10:04.242072 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:10:04.242074 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:10:04.242125 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:10:04.242466 (+ 341us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:10:04.242470 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:10:04.242472 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:10:04.242665 (+ 193us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:10:04.242771 (+ 106us) server_negotiation.cc:1036] Waiting for connection context
0903 08:10:04.243566 (+ 795us) server_negotiation.cc:300] Negotiation successful
0903 08:10:04.243715 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20250903 08:10:04.245262 25558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:10:04.228112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:44561 (local address 127.12.201.195:39397)
0903 08:10:04.229923 (+ 1811us) negotiation.cc:107] Waiting for socket to connect
0903 08:10:04.229970 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0903 08:10:04.230960 (+ 990us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:10:04.231768 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:10:04.231780 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:10:04.232266 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:10:04.233066 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:10:04.233084 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:10:04.234240 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:10:04.234243 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:10:04.234883 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:10:04.234892 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:10:04.235474 (+ 582us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:10:04.236833 (+ 1359us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:10:04.236869 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:10:04.239210 (+ 2341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:10:04.241561 (+ 2351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:10:04.241567 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:10:04.241582 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:10:04.241937 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:10:04.242235 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:10:04.242239 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:10:04.242241 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:10:04.242364 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:10:04.242863 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:10:04.242870 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:10:04.243242 (+ 372us) client_negotiation.cc:769] Sending connection context
0903 08:10:04.244135 (+ 893us) client_negotiation.cc:240] Negotiation successful
0903 08:10:04.244408 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1608,"thread_start_us":137,"threads_started":1}
I20250903 08:10:04.246485 25556 heartbeater.cc:344] Connected to a master server at 127.12.201.254:44561
I20250903 08:10:04.246760 25556 heartbeater.cc:461] Registering TS with master...
I20250903 08:10:04.247422 25556 heartbeater.cc:507] Master 127.12.201.254:44561 requested a full tablet report, sending...
I20250903 08:10:04.249554 25067 ts_manager.cc:194] Registered new tserver with Master: d9d4d7336d5e455999d645721b3a6c0c (127.12.201.195:40721)
I20250903 08:10:04.250304 25067 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.201.195@KRBTEST.COM'} at 127.12.201.195:39397
I20250903 08:10:04.261307 13095 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250903 08:10:04.273109 25284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:10:04.263538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38512 (local address 127.12.201.254:44561)
0903 08:10:04.263672 (+ 134us) server_negotiation.cc:207] Beginning negotiation
0903 08:10:04.263677 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 08:10:04.265534 (+ 1857us) server_negotiation.cc:408] Connection header received
0903 08:10:04.265676 (+ 142us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:10:04.265679 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:10:04.265756 (+ 77us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:10:04.265835 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:10:04.266858 (+ 1023us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:10:04.267417 (+ 559us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:10:04.268105 (+ 688us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:10:04.268322 (+ 217us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:10:04.269485 (+ 1163us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:10:04.269504 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:10:04.269507 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:10:04.269532 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:10:04.271396 (+ 1864us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:10:04.271891 (+ 495us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:10:04.271895 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:10:04.271897 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:10:04.271952 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:10:04.272203 (+ 251us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:10:04.272208 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:10:04.272213 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:10:04.272378 (+ 165us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:10:04.272469 (+ 91us) server_negotiation.cc:1036] Waiting for connection context
0903 08:10:04.272768 (+ 299us) server_negotiation.cc:300] Negotiation successful
0903 08:10:04.272892 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20250903 08:10:04.275992 25067 catalog_manager.cc:2232] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38512:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 08:10:04.279325 25067 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
Sep 03 08:10:04 dist-test-slave-27r9 krb5kdc[24564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756887003, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.254@KRBTEST.COM for HTTP/127.12.201.212@KRBTEST.COM
I20250903 08:10:04.296698 25570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:10:04.290343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.195:40721 (local address 127.0.0.1:44174)
0903 08:10:04.291336 (+ 993us) negotiation.cc:107] Waiting for socket to connect
0903 08:10:04.291377 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0903 08:10:04.291511 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:10:04.292620 (+ 1109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:10:04.292626 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:10:04.292647 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 08:10:04.292923 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:10:04.292930 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:10:04.295280 (+ 2350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:10:04.295285 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:10:04.296191 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:10:04.296202 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:10:04.296318 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:10:04.296374 (+ 56us) client_negotiation.cc:769] Sending connection context
0903 08:10:04.296464 (+ 90us) client_negotiation.cc:240] Negotiation successful
0903 08:10:04.296555 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":820,"thread_start_us":115,"threads_started":1}
I20250903 08:10:04.297914 25571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:10:04.291223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44174 (local address 127.12.201.195:40721)
0903 08:10:04.292187 (+ 964us) server_negotiation.cc:207] Beginning negotiation
0903 08:10:04.292192 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 08:10:04.292211 (+ 19us) server_negotiation.cc:408] Connection header received
0903 08:10:04.292278 (+ 67us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:10:04.292282 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:10:04.292452 (+ 170us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:10:04.293045 (+ 593us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 08:10:04.293153 (+ 108us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:10:04.295137 (+ 1984us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:10:04.297005 (+ 1868us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:10:04.297499 (+ 494us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:10:04.297607 (+ 108us) server_negotiation.cc:1036] Waiting for connection context
0903 08:10:04.297668 (+ 61us) server_negotiation.cc:300] Negotiation successful
0903 08:10:04.297778 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":880,"thread_start_us":89,"threads_started":1}
I20250903 08:10:04.298300 25572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:10:04.290729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.193:44041 (local address 127.0.0.1:48488)
0903 08:10:04.291881 (+ 1152us) negotiation.cc:107] Waiting for socket to connect
0903 08:10:04.291895 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0903 08:10:04.291969 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:10:04.294749 (+ 2780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:10:04.294753 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:10:04.294772 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 08:10:04.295071 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:10:04.295078 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:10:04.297101 (+ 2023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:10:04.297107 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:10:04.297966 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:10:04.297974 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:10:04.298062 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:10:04.298078 (+ 16us) client_negotiation.cc:769] Sending connection context
0903 08:10:04.298113 (+ 35us) client_negotiation.cc:240] Negotiation successful
0903 08:10:04.298182 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1075,"thread_start_us":94,"threads_started":1}
I20250903 08:10:04.300261 25569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:10:04.291659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.194:43813 (local address 127.0.0.1:47402)
0903 08:10:04.294885 (+ 3226us) negotiation.cc:107] Waiting for socket to connect
0903 08:10:04.294898 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0903 08:10:04.294972 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:10:04.295528 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:10:04.295531 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:10:04.295547 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 08:10:04.295801 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:10:04.295811 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:10:04.299012 (+ 3201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:10:04.299017 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:10:04.299902 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:10:04.299912 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:10:04.300034 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:10:04.300048 (+ 14us) client_negotiation.cc:769] Sending connection context
0903 08:10:04.300094 (+ 46us) client_negotiation.cc:240] Negotiation successful
0903 08:10:04.300144 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3161,"thread_start_us":80,"threads_started":1}
I20250903 08:10:04.300776 25490 tablet_service.cc:1468] Processing CreateTablet for tablet 4603e5a56ad74621b4730c9e6e0c8d0e (DEFAULT_TABLE table=test-table [id=063b167588ce409d8fd2858f2b175797]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:10:04.301599 25574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:10:04.293805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47402 (local address 127.12.201.194:43813)
0903 08:10:04.294669 (+ 864us) server_negotiation.cc:207] Beginning negotiation
0903 08:10:04.294675 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0903 08:10:04.295160 (+ 485us) server_negotiation.cc:408] Connection header received
0903 08:10:04.295254 (+ 94us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:10:04.295261 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:10:04.295399 (+ 138us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:10:04.295528 (+ 129us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 08:10:04.295936 (+ 408us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:10:04.298861 (+ 2925us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:10:04.300495 (+ 1634us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:10:04.301059 (+ 564us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:10:04.301197 (+ 138us) server_negotiation.cc:1036] Waiting for connection context
0903 08:10:04.301392 (+ 195us) server_negotiation.cc:300] Negotiation successful
0903 08:10:04.301479 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":763,"thread_start_us":80,"threads_started":1}
I20250903 08:10:04.302488 25490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4603e5a56ad74621b4730c9e6e0c8d0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:10:04.304349 25354 tablet_service.cc:1468] Processing CreateTablet for tablet 4603e5a56ad74621b4730c9e6e0c8d0e (DEFAULT_TABLE table=test-table [id=063b167588ce409d8fd2858f2b175797]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:10:04.305378 25354 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4603e5a56ad74621b4730c9e6e0c8d0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:10:04.310781 25573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:10:04.293523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48488 (local address 127.12.201.193:44041)
0903 08:10:04.294117 (+ 594us) server_negotiation.cc:207] Beginning negotiation
0903 08:10:04.294123 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0903 08:10:04.294147 (+ 24us) server_negotiation.cc:408] Connection header received
0903 08:10:04.294231 (+ 84us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:10:04.294239 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:10:04.294414 (+ 175us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:10:04.295702 (+ 1288us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 08:10:04.295827 (+ 125us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:10:04.296951 (+ 1124us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:10:04.298597 (+ 1646us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:10:04.310358 (+ 11761us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:10:04.310472 (+ 114us) server_negotiation.cc:1036] Waiting for connection context
0903 08:10:04.310545 (+ 73us) server_negotiation.cc:300] Negotiation successful
0903 08:10:04.310638 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":83,"threads_started":1}
I20250903 08:10:04.312598 25575 tablet_bootstrap.cc:492] T 4603e5a56ad74621b4730c9e6e0c8d0e P d9d4d7336d5e455999d645721b3a6c0c: Bootstrap starting.
I20250903 08:10:04.314929 25575 tablet_bootstrap.cc:654] T 4603e5a56ad74621b4730c9e6e0c8d0e P d9d4d7336d5e455999d645721b3a6c0c: Neither blocks nor log segments found. Creating new log.
I20250903 08:10:04.316856 25575 log.cc:826] T 4603e5a56ad74621b4730c9e6e0c8d0e P d9d4d7336d5e455999d645721b3a6c0c: Log is configured to *not* fsync() on all Append() calls
I20250903 08:10:04.317128 25216 tablet_service.cc:1468] Processing CreateTablet for tablet 4603e5a56ad74621b4730c9e6e0c8d0e (DEFAULT_TABLE table=test-table [id=063b167588ce409d8fd2858f2b175797]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:10:04.318212 25216 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4603e5a56ad74621b4730c9e6e0c8d0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:10:04.323756 25577 tablet_bootstrap.cc:492] T 4603e5a56ad74621b4730c9e6e0c8d0e P 6492ed940b264ca4bccc57bf3b00e4ea: Bootstrap starting.
I20250903 08:10:04.326114 25577 tablet_bootstrap.cc:654] T 4603e5a56ad74621b4730c9e6e0c8d0e P 6492ed940b264ca4bccc57bf3b00e4ea: Neither blocks nor log segments found. Creating new log.
I20250903 08:10:04.327054 25577 log.cc:826] T 4603e5a56ad74621b4730c9e6e0c8d0e P 6492ed940b264ca4bccc57bf3b00e4ea: Log is configured to *not* fsync() on all Append() calls
I20250903 08:10:04.328680 25575 tablet_bootstrap.cc:492] T 4603e5a56ad74621b4730c9e6e0c8d0e P d9d4d7336d5e455999d645721b3a6c0c: No bootstrap required, opened a new log
I20250903 08:10:04.328943 25575 ts_tablet_manager.cc:1397] T 4603e5a56ad74621b4730c9e6e0c8d0e P d9d4d7336d5e455999d645721b3a6c0c: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.000s
I20250903 08:10:04.332571 25575 raft_consensus.cc:357] T 4603e5a56ad74621b4730c9e6e0c8d0e P d9d4d7336d5e455999d645721b3a6c0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d4d7336d5e455999d645721b3a6c0c" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 40721 } } peers { permanent_uuid: "39beab15348d4453bde7185836c0de2c" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 43813 } } peers { permanent_uuid: "6492ed940b264ca4bccc57bf3b00e4ea" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 44041 } }
I20250903 08:10:04.332947 25575 raft_consensus.cc:383] T 4603e5a56ad74621b4730c9e6e0c8d0e P d9d4d7336d5e455999d645721b3a6c0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:10:04.333053 25575 raft_consensus.cc:738] T 4603e5a56ad74621b4730c9e6e0c8d0e P d9d4d7336d5e455999d645721b3a6c0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9d4d7336d5e455999d645721b3a6c0c, State: Initialized, Role: FOLLOWER
I20250903 08:10:04.333598 25575 consensus_queue.cc:260] T 4603e5a56ad74621b4730c9e6e0c8d0e P d9d4d7336d5e455999d645721b3a6c0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d4d7336d5e455999d645721b3a6c0c" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 40721 } } peers { permanent_uuid: "39beab15348d4453bde7185836c0de2c" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 43813 } } peers { permanent_uuid: "6492ed940b264ca4bccc57bf3b00e4ea" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 44041 } }
I20250903 08:10:04.334594 25575 ts_tablet_manager.cc:1428] T 4603e5a56ad74621b4730c9e6e0c8d0e P d9d4d7336d5e455999d645721b3a6c0c: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250903 08:10:04.335048 25556 heartbeater.cc:499] Master 127.12.201.254:44561 was elected leader, sending a full tablet report...
I20250903 08:10:04.335363 25580 tablet_bootstrap.cc:492] T 4603e5a56ad74621b4730c9e6e0c8d0e P 39beab15348d4453bde7185836c0de2c: Bootstrap starting.
I20250903 08:10:04.337785 25580 tablet_bootstrap.cc:654] T 4603e5a56ad74621b4730c9e6e0c8d0e P 39beab15348d4453bde7185836c0de2c: Neither blocks nor log segments found. Creating new log.
I20250903 08:10:04.338519 25580 log.cc:826] T 4603e5a56ad74621b4730c9e6e0c8d0e P 39beab15348d4453bde7185836c0de2c: Log is configured to *not* fsync() on all Append() calls
I20250903 08:10:04.344777 25577 tablet_bootstrap.cc:492] T 4603e5a56ad74621b4730c9e6e0c8d0e P 6492ed940b264ca4bccc57bf3b00e4ea: No bootstrap required, opened a new log
I20250903 08:10:04.345026 25577 ts_tablet_manager.cc:1397] T 4603e5a56ad74621b4730c9e6e0c8d0e P 6492ed940b264ca4bccc57bf3b00e4ea: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.000s
I20250903 08:10:04.345352 25580 tablet_bootstrap.cc:492] T 4603e5a56ad74621b4730c9e6e0c8d0e P 39beab15348d4453bde7185836c0de2c: No bootstrap required, opened a new log
I20250903 08:10:04.345562 25580 ts_tablet_manager.cc:1397] T 4603e5a56ad74621b4730c9e6e0c8d0e P 39beab15348d4453bde7185836c0de2c: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20250903 08:10:04.348744 25577 raft_consensus.cc:357] T 4603e5a56ad74621b4730c9e6e0c8d0e P 6492ed940b264ca4bccc57bf3b00e4ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d4d7336d5e455999d645721b3a6c0c" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 40721 } } peers { permanent_uuid: "39beab15348d4453bde7185836c0de2c" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 43813 } } peers { permanent_uuid: "6492ed940b264ca4bccc57bf3b00e4ea" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 44041 } }
I20250903 08:10:04.348747 25580 raft_consensus.cc:357] T 4603e5a56ad74621b4730c9e6e0c8d0e P 39beab15348d4453bde7185836c0de2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d4d7336d5e455999d645721b3a6c0c" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 40721 } } peers { permanent_uuid: "39beab15348d4453bde7185836c0de2c" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 43813 } } peers { permanent_uuid: "6492ed940b264ca4bccc57bf3b00e4ea" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 44041 } }
I20250903 08:10:04.349005 25577 raft_consensus.cc:383] T 4603e5a56ad74621b4730c9e6e0c8d0e P 6492ed940b264ca4bccc57bf3b00e4ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:10:04.349086 25577 raft_consensus.cc:738] T 4603e5a56ad74621b4730c9e6e0c8d0e P 6492ed940b264ca4bccc57bf3b00e4ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6492ed940b264ca4bccc57bf3b00e4ea, State: Initialized, Role: FOLLOWER
I20250903 08:10:04.349170 25580 raft_consensus.cc:383] T 4603e5a56ad74621b4730c9e6e0c8d0e P 39beab15348d4453bde7185836c0de2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:10:04.349269 25580 raft_consensus.cc:738] T 4603e5a56ad74621b4730c9e6e0c8d0e P 39beab15348d4453bde7185836c0de2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39beab15348d4453bde7185836c0de2c, State: Initialized, Role: FOLLOWER
I20250903 08:10:04.349555 25577 consensus_queue.cc:260] T 4603e5a56ad74621b4730c9e6e0c8d0e P 6492ed940b264ca4bccc57bf3b00e4ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d4d7336d5e455999d645721b3a6c0c" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 40721 } } peers { permanent_uuid: "39beab15348d4453bde7185836c0de2c" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 43813 } } peers { permanent_uuid: "6492ed940b264ca4bccc57bf3b00e4ea" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 44041 } }
I20250903 08:10:04.349828 25580 consensus_queue.cc:260] T 4603e5a56ad74621b4730c9e6e0c8d0e P 39beab15348d4453bde7185836c0de2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d4d7336d5e455999d645721b3a6c0c" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 40721 } } peers { permanent_uuid: "39beab15348d4453bde7185836c0de2c" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 43813 } } peers { permanent_uuid: "6492ed940b264ca4bccc57bf3b00e4ea" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 44041 } }
I20250903 08:10:04.350515 25577 ts_tablet_manager.cc:1428] T 4603e5a56ad74621b4730c9e6e0c8d0e P 6492ed940b264ca4bccc57bf3b00e4ea: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20250903 08:10:04.351016 25282 heartbeater.cc:499] Master 127.12.201.254:44561 was elected leader, sending a full tablet report...
I20250903 08:10:04.352792 25580 ts_tablet_manager.cc:1428] T 4603e5a56ad74621b4730c9e6e0c8d0e P 39beab15348d4453bde7185836c0de2c: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20250903 08:10:04.353304 25420 heartbeater.cc:499] Master 127.12.201.254:44561 was elected leader, sending a full tablet report...
I20250903 08:10:04.398002 25584 raft_consensus.cc:491] T 4603e5a56ad74621b4730c9e6e0c8d0e P 39beab15348d4453bde7185836c0de2c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 08:10:04.398234 25584 raft_consensus.cc:513] T 4603e5a56ad74621b4730c9e6e0c8d0e P 39beab15348d4453bde7185836c0de2c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d4d7336d5e455999d645721b3a6c0c" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 40721 } } peers { permanent_uuid: "39beab15348d4453bde7185836c0de2c" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 43813 } } peers { permanent_uuid: "6492ed940b264ca4bccc57bf3b00e4ea" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 44041 } }
I20250903 08:10:04.399644 25584 leader_election.cc:290] T 4603e5a56ad74621b4730c9e6e0c8d0e P 39beab15348d4453bde7185836c0de2c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9d4d7336d5e455999d645721b3a6c0c (127.12.201.195:40721), 6492ed940b264ca4bccc57bf3b00e4ea (127.12.201.193:44041)
I20250903 08:10:04.408106 25571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:10:04.400362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.194:44221 (local address 127.12.201.195:40721)
0903 08:10:04.400511 (+ 149us) server_negotiation.cc:207] Beginning negotiation
0903 08:10:04.400517 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0903 08:10:04.403944 (+ 3427us) server_negotiation.cc:408] Connection header received
0903 08:10:04.404212 (+ 268us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:10:04.404219 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:10:04.404285 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:10:04.404339 (+ 54us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 08:10:04.404983 (+ 644us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:10:04.405796 (+ 813us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:10:04.407090 (+ 1294us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:10:04.407585 (+ 495us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:10:04.407633 (+ 48us) server_negotiation.cc:1036] Waiting for connection context
0903 08:10:04.407864 (+ 231us) server_negotiation.cc:300] Negotiation successful
0903 08:10:04.407943 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20250903 08:10:04.408179 25573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:10:04.401113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.194:40487 (local address 127.12.201.193:44041)
0903 08:10:04.401237 (+ 124us) server_negotiation.cc:207] Beginning negotiation
0903 08:10:04.401241 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0903 08:10:04.404276 (+ 3035us) server_negotiation.cc:408] Connection header received
0903 08:10:04.404476 (+ 200us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:10:04.404480 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:10:04.404531 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:10:04.404578 (+ 47us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 08:10:04.405198 (+ 620us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:10:04.405984 (+ 786us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:10:04.407232 (+ 1248us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:10:04.407734 (+ 502us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:10:04.407772 (+ 38us) server_negotiation.cc:1036] Waiting for connection context
0903 08:10:04.407990 (+ 218us) server_negotiation.cc:300] Negotiation successful
0903 08:10:04.408056 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20250903 08:10:04.408557 25585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:10:04.401034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.193:44041 (local address 127.12.201.194:40487)
0903 08:10:04.404175 (+ 3141us) negotiation.cc:107] Waiting for socket to connect
0903 08:10:04.404190 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0903 08:10:04.404352 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:10:04.404710 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:10:04.404713 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:10:04.404732 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 08:10:04.405055 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:10:04.405063 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:10:04.406126 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:10:04.406129 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:10:04.407095 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:10:04.407106 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:10:04.407834 (+ 728us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:10:04.407850 (+ 16us) client_negotiation.cc:769] Sending connection context
0903 08:10:04.408327 (+ 477us) client_negotiation.cc:240] Negotiation successful
0903 08:10:04.408393 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3065,"thread_start_us":98,"threads_started":1}
I20250903 08:10:04.408618 25586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:10:04.400038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.195:40721 (local address 127.12.201.194:44221)
0903 08:10:04.403837 (+ 3799us) negotiation.cc:107] Waiting for socket to connect
0903 08:10:04.403857 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0903 08:10:04.404060 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:10:04.404489 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:10:04.404493 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:10:04.404513 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0903 08:10:04.404838 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:10:04.404848 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:10:04.405940 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:10:04.405943 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:10:04.406892 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:10:04.406903 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:10:04.407706 (+ 803us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:10:04.407724 (+ 18us) client_negotiation.cc:769] Sending connection context
0903 08:10:04.408286 (+ 562us) client_negotiation.cc:240] Negotiation successful
0903 08:10:04.408356 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3706,"thread_start_us":640,"threads_started":1}
I20250903 08:10:04.409242 25236 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4603e5a56ad74621b4730c9e6e0c8d0e" candidate_uuid: "39beab15348d4453bde7185836c0de2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6492ed940b264ca4bccc57bf3b00e4ea" is_pre_election: true
I20250903 08:10:04.409242 25495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4603e5a56ad74621b4730c9e6e0c8d0e" candidate_uuid: "39beab15348d4453bde7185836c0de2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9d4d7336d5e455999d645721b3a6c0c" is_pre_election: true
I20250903 08:10:04.409539 25236 raft_consensus.cc:2466] T 4603e5a56ad74621b4730c9e6e0c8d0e P 6492ed940b264ca4bccc57bf3b00e4ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39beab15348d4453bde7185836c0de2c in term 0.
I20250903 08:10:04.409540 25495 raft_consensus.cc:2466] T 4603e5a56ad74621b4730c9e6e0c8d0e P d9d4d7336d5e455999d645721b3a6c0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39beab15348d4453bde7185836c0de2c in term 0.
I20250903 08:10:04.410037 25307 leader_election.cc:304] T 4603e5a56ad74621b4730c9e6e0c8d0e P 39beab15348d4453bde7185836c0de2c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39beab15348d4453bde7185836c0de2c, d9d4d7336d5e455999d645721b3a6c0c; no voters:
I20250903 08:10:04.410347 25584 raft_consensus.cc:2802] T 4603e5a56ad74621b4730c9e6e0c8d0e P 39beab15348d4453bde7185836c0de2c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 08:10:04.410430 25584 raft_consensus.cc:491] T 4603e5a56ad74621b4730c9e6e0c8d0e P 39beab15348d4453bde7185836c0de2c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 08:10:04.410480 25584 raft_consensus.cc:3058] T 4603e5a56ad74621b4730c9e6e0c8d0e P 39beab15348d4453bde7185836c0de2c [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:10:04.411772 25584 raft_consensus.cc:513] T 4603e5a56ad74621b4730c9e6e0c8d0e P 39beab15348d4453bde7185836c0de2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d4d7336d5e455999d645721b3a6c0c" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 40721 } } peers { permanent_uuid: "39beab15348d4453bde7185836c0de2c" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 43813 } } peers { permanent_uuid: "6492ed940b264ca4bccc57bf3b00e4ea" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 44041 } }
I20250903 08:10:04.412189 25584 leader_election.cc:290] T 4603e5a56ad74621b4730c9e6e0c8d0e P 39beab15348d4453bde7185836c0de2c [CANDIDATE]: Term 1 election: Requested vote from peers d9d4d7336d5e455999d645721b3a6c0c (127.12.201.195:40721), 6492ed940b264ca4bccc57bf3b00e4ea (127.12.201.193:44041)
I20250903 08:10:04.412529 25236 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4603e5a56ad74621b4730c9e6e0c8d0e" candidate_uuid: "39beab15348d4453bde7185836c0de2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6492ed940b264ca4bccc57bf3b00e4ea"
I20250903 08:10:04.412653 25236 raft_consensus.cc:3058] T 4603e5a56ad74621b4730c9e6e0c8d0e P 6492ed940b264ca4bccc57bf3b00e4ea [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:10:04.412644 25495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4603e5a56ad74621b4730c9e6e0c8d0e" candidate_uuid: "39beab15348d4453bde7185836c0de2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9d4d7336d5e455999d645721b3a6c0c"
I20250903 08:10:04.412757 25495 raft_consensus.cc:3058] T 4603e5a56ad74621b4730c9e6e0c8d0e P d9d4d7336d5e455999d645721b3a6c0c [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:10:04.413916 25495 raft_consensus.cc:2466] T 4603e5a56ad74621b4730c9e6e0c8d0e P d9d4d7336d5e455999d645721b3a6c0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39beab15348d4453bde7185836c0de2c in term 1.
I20250903 08:10:04.413920 25236 raft_consensus.cc:2466] T 4603e5a56ad74621b4730c9e6e0c8d0e P 6492ed940b264ca4bccc57bf3b00e4ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39beab15348d4453bde7185836c0de2c in term 1.
I20250903 08:10:04.414273 25307 leader_election.cc:304] T 4603e5a56ad74621b4730c9e6e0c8d0e P 39beab15348d4453bde7185836c0de2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39beab15348d4453bde7185836c0de2c, d9d4d7336d5e455999d645721b3a6c0c; no voters:
I20250903 08:10:04.414522 25584 raft_consensus.cc:2802] T 4603e5a56ad74621b4730c9e6e0c8d0e P 39beab15348d4453bde7185836c0de2c [term 1 FOLLOWER]: Leader election won for term 1
I20250903 08:10:04.414742 25584 raft_consensus.cc:695] T 4603e5a56ad74621b4730c9e6e0c8d0e P 39beab15348d4453bde7185836c0de2c [term 1 LEADER]: Becoming Leader. State: Replica: 39beab15348d4453bde7185836c0de2c, State: Running, Role: LEADER
I20250903 08:10:04.415120 25584 consensus_queue.cc:237] T 4603e5a56ad74621b4730c9e6e0c8d0e P 39beab15348d4453bde7185836c0de2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d4d7336d5e455999d645721b3a6c0c" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 40721 } } peers { permanent_uuid: "39beab15348d4453bde7185836c0de2c" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 43813 } } peers { permanent_uuid: "6492ed940b264ca4bccc57bf3b00e4ea" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 44041 } }
I20250903 08:10:04.418618 25067 catalog_manager.cc:5582] T 4603e5a56ad74621b4730c9e6e0c8d0e P 39beab15348d4453bde7185836c0de2c reported cstate change: term changed from 0 to 1, leader changed from <none> to 39beab15348d4453bde7185836c0de2c (127.12.201.194). New cstate: current_term: 1 leader_uuid: "39beab15348d4453bde7185836c0de2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d4d7336d5e455999d645721b3a6c0c" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 40721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39beab15348d4453bde7185836c0de2c" member_type: VOTER last_known_addr { host: "127.12.201.194" port: 43813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6492ed940b264ca4bccc57bf3b00e4ea" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 44041 } health_report { overall_health: UNKNOWN } } }
I20250903 08:10:04.454169 25574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:10:04.449495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47410 (local address 127.12.201.194:43813)
0903 08:10:04.449627 (+ 132us) server_negotiation.cc:207] Beginning negotiation
0903 08:10:04.449632 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0903 08:10:04.449648 (+ 16us) server_negotiation.cc:408] Connection header received
0903 08:10:04.449872 (+ 224us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:10:04.449876 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:10:04.449946 (+ 70us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:10:04.450049 (+ 103us) server_negotiation.cc:227] Negotiated authn=TOKEN
0903 08:10:04.450619 (+ 570us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:10:04.451446 (+ 827us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:10:04.452504 (+ 1058us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:10:04.452720 (+ 216us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:10:04.452843 (+ 123us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0903 08:10:04.453415 (+ 572us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0903 08:10:04.453548 (+ 133us) server_negotiation.cc:1036] Waiting for connection context
0903 08:10:04.453965 (+ 417us) server_negotiation.cc:300] Negotiation successful
0903 08:10:04.454028 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20250903 08:10:04.461067 25236 raft_consensus.cc:1273] T 4603e5a56ad74621b4730c9e6e0c8d0e P 6492ed940b264ca4bccc57bf3b00e4ea [term 1 FOLLOWER]: Refusing update from remote peer 39beab15348d4453bde7185836c0de2c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 08:10:04.461844 25584 consensus_queue.cc:1035] T 4603e5a56ad74621b4730c9e6e0c8d0e P 39beab15348d4453bde7185836c0de2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6492ed940b264ca4bccc57bf3b00e4ea" member_type: VOTER last_known_addr { host: "127.12.201.193" port: 44041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250903 08:10:04.469211 25283 tablet.cc:2378] T 4603e5a56ad74621b4730c9e6e0c8d0e P 6492ed940b264ca4bccc57bf3b00e4ea: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250903 08:10:04.479434 25495 raft_consensus.cc:1273] T 4603e5a56ad74621b4730c9e6e0c8d0e P d9d4d7336d5e455999d645721b3a6c0c [term 1 FOLLOWER]: Refusing update from remote peer 39beab15348d4453bde7185836c0de2c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250903 08:10:04.479825 25557 tablet.cc:2378] T 4603e5a56ad74621b4730c9e6e0c8d0e P d9d4d7336d5e455999d645721b3a6c0c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250903 08:10:04.480186 25584 consensus_queue.cc:1035] T 4603e5a56ad74621b4730c9e6e0c8d0e P 39beab15348d4453bde7185836c0de2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9d4d7336d5e455999d645721b3a6c0c" member_type: VOTER last_known_addr { host: "127.12.201.195" port: 40721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 08:10:04.487556 25590 mvcc.cc:204] Tried to move back new op lower bound from 7196209170264113152 to 7196209170087649280. Current Snapshot: MvccSnapshot[applied={T|T < 7196209170264113152}]
I20250903 08:10:04.490867 25591 mvcc.cc:204] Tried to move back new op lower bound from 7196209170264113152 to 7196209170087649280. Current Snapshot: MvccSnapshot[applied={T|T < 7196209170264113152}]
I20250903 08:10:04.495494 25593 mvcc.cc:204] Tried to move back new op lower bound from 7196209170264113152 to 7196209170087649280. Current Snapshot: MvccSnapshot[applied={T|T < 7196209170264113152}]
I20250903 08:10:04.504441 25067 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38512:
table { table_name: "test-table" } modify_external_catalogs: true
I20250903 08:10:04.504630 25067 catalog_manager.cc:2730] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38512:
table { table_name: "test-table" } modify_external_catalogs: true
I20250903 08:10:04.507344 25067 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 61b76a5beaa947ea8703e0e50b0e3de4: Sending DeleteTablet for 3 replicas of tablet 4603e5a56ad74621b4730c9e6e0c8d0e
I20250903 08:10:04.508301 25216 tablet_service.cc:1515] Processing DeleteTablet for tablet 4603e5a56ad74621b4730c9e6e0c8d0e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 08:10:04 UTC) from {username='kudu', principal='kudu/127.12.201.254@KRBTEST.COM'} at 127.0.0.1:48488
I20250903 08:10:04.509398 25490 tablet_service.cc:1515] Processing DeleteTablet for tablet 4603e5a56ad74621b4730c9e6e0c8d0e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 08:10:04 UTC) from {username='kudu', principal='kudu/127.12.201.254@KRBTEST.COM'} at 127.0.0.1:44174
I20250903 08:10:04.509871 25354 tablet_service.cc:1515] Processing DeleteTablet for tablet 4603e5a56ad74621b4730c9e6e0c8d0e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 08:10:04 UTC) from {username='kudu', principal='kudu/127.12.201.254@KRBTEST.COM'} at 127.0.0.1:47402
I20250903 08:10:04.510998 13095 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUML90r/build/debug/bin/kudu with pid 25149
I20250903 08:10:04.514873 25600 tablet_replica.cc:331] T 4603e5a56ad74621b4730c9e6e0c8d0e P d9d4d7336d5e455999d645721b3a6c0c: stopping tablet replica
I20250903 08:10:04.515233 25600 raft_consensus.cc:2241] T 4603e5a56ad74621b4730c9e6e0c8d0e P d9d4d7336d5e455999d645721b3a6c0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 08:10:04.515642 25600 raft_consensus.cc:2270] T 4603e5a56ad74621b4730c9e6e0c8d0e P d9d4d7336d5e455999d645721b3a6c0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 08:10:04.515865 25601 tablet_replica.cc:331] T 4603e5a56ad74621b4730c9e6e0c8d0e P 39beab15348d4453bde7185836c0de2c: stopping tablet replica
I20250903 08:10:04.516112 25601 raft_consensus.cc:2241] T 4603e5a56ad74621b4730c9e6e0c8d0e P 39beab15348d4453bde7185836c0de2c [term 1 LEADER]: Raft consensus shutting down.
I20250903 08:10:04.516592 25601 raft_consensus.cc:2270] T 4603e5a56ad74621b4730c9e6e0c8d0e P 39beab15348d4453bde7185836c0de2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 08:10:04.518096 25601 ts_tablet_manager.cc:1905] T 4603e5a56ad74621b4730c9e6e0c8d0e P 39beab15348d4453bde7185836c0de2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 08:10:04.520984 25601 ts_tablet_manager.cc:1918] T 4603e5a56ad74621b4730c9e6e0c8d0e P 39beab15348d4453bde7185836c0de2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250903 08:10:04.521085 25601 log.cc:1199] T 4603e5a56ad74621b4730c9e6e0c8d0e P 39beab15348d4453bde7185836c0de2c: Deleting WAL directory at /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/minicluster-data/ts-1/wal/wals/4603e5a56ad74621b4730c9e6e0c8d0e
W20250903 08:10:04.521646 25054 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.201.193:44041 (error 108)
I20250903 08:10:04.522310 25601 ts_tablet_manager.cc:1939] T 4603e5a56ad74621b4730c9e6e0c8d0e P 39beab15348d4453bde7185836c0de2c: Deleting consensus metadata
I20250903 08:10:04.522778 25600 ts_tablet_manager.cc:1905] T 4603e5a56ad74621b4730c9e6e0c8d0e P d9d4d7336d5e455999d645721b3a6c0c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 08:10:04.523494 25051 catalog_manager.cc:4928] TS 39beab15348d4453bde7185836c0de2c (127.12.201.194:43813): tablet 4603e5a56ad74621b4730c9e6e0c8d0e (table test-table [id=063b167588ce409d8fd2858f2b175797]) successfully deleted
I20250903 08:10:04.524470 25569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:10:04.524151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.193:44041 (local address 127.0.0.1:48490)
0903 08:10:04.524327 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0903 08:10:04.524384 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.201.193:44041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
W20250903 08:10:04.524626 25054 catalog_manager.cc:4671] TS 6492ed940b264ca4bccc57bf3b00e4ea (127.12.201.193:44041): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4603e5a56ad74621b4730c9e6e0c8d0e: Network error: Client connection negotiation failed: client connection to 127.12.201.193:44041: connect: Connection refused (error 111)
I20250903 08:10:04.525394 25600 ts_tablet_manager.cc:1918] T 4603e5a56ad74621b4730c9e6e0c8d0e P d9d4d7336d5e455999d645721b3a6c0c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250903 08:10:04.525489 25600 log.cc:1199] T 4603e5a56ad74621b4730c9e6e0c8d0e P d9d4d7336d5e455999d645721b3a6c0c: Deleting WAL directory at /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756886860253814-13095-0/minicluster-data/ts-2/wal/wals/4603e5a56ad74621b4730c9e6e0c8d0e
I20250903 08:10:04.525853 25600 ts_tablet_manager.cc:1939] T 4603e5a56ad74621b4730c9e6e0c8d0e P d9d4d7336d5e455999d645721b3a6c0c: Deleting consensus metadata
I20250903 08:10:04.527189 25051 catalog_manager.cc:4928] TS d9d4d7336d5e455999d645721b3a6c0c (127.12.201.195:40721): tablet 4603e5a56ad74621b4730c9e6e0c8d0e (table test-table [id=063b167588ce409d8fd2858f2b175797]) successfully deleted
I20250903 08:10:04.527583 13095 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUML90r/build/debug/bin/kudu with pid 25290
I20250903 08:10:04.536651 13095 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUML90r/build/debug/bin/kudu with pid 25427
I20250903 08:10:04.544294 13095 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUML90r/build/debug/bin/kudu with pid 25035
I20250903 08:10:04.550099 13095 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20250903 08:10:04.895212 13095 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20250903 08:10:04.895342 13095 mini_ranger.cc:66] Stopping Ranger...
I20250903 08:10:05.231315 13095 mini_ranger.cc:68] Stopped Ranger
2025-09-03 08:10:05.231 UTC [24595] LOG: received smart shutdown request
2025-09-03 08:10:05.235 UTC [24595] LOG: background worker "logical replication launcher" (PID 24603) exited with exit code 1
2025-09-03 08:10:05.235 UTC [24598] LOG: shutting down
2025-09-03 08:10:05.237 UTC [24598] LOG: checkpoint starting: shutdown immediate
2025-09-03 08:10:06.179 UTC [24598] LOG: checkpoint complete: wrote 2411 buffers (14.7%); 0 WAL file(s) added, 0 removed, 1 recycled; write=0.036 s, sync=0.900 s, total=0.944 s; sync files=1108, longest=0.006 s, average=0.001 s; distance=12073 kB, estimate=12073 kB; lsn=0/20450C8, redo lsn=0/20450C8
2025-09-03 08:10:06.187 UTC [24595] LOG: database system is shut down
2025-09-03T08:10:06Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (49502 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 03 08:10:06 dist-test-slave-27r9 krb5kdc[25611](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 03 08:10:06 dist-test-slave-27r9 krb5kdc[25611](info): set up 2 sockets
Sep 03 08:10:06 dist-test-slave-27r9 krb5kdc[25611](info): commencing operation
krb5kdc: starting...
W20250903 08:10:08.291322 13095 mini_kdc.cc:121] Time spent starting KDC: real 1.956s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/krb5kdc/test-user.keytab.
Sep 03 08:10:08 dist-test-slave-27r9 krb5kdc[25611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756887008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-09-03T08:10:08Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-03T08:10:08Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.12.201.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.12.201.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.12.201.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/krb5kdc/rangeradmin_127.12.201.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.12.201.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/krb5kdc/rangeradmin_127.12.201.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.12.201.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.12.201.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.12.201.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/krb5kdc/rangerlookup_127.12.201.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.12.201.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/krb5kdc/rangerlookup_127.12.201.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.12.201.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.201.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.201.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/krb5kdc/HTTP_127.12.201.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.12.201.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/krb5kdc/HTTP_127.12.201.212@KRBTEST.COM.keytab.
I20250903 08:10:08.445206 13095 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default "max_connections" ... 100
selecting default "shared_buffers" ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskUML90r/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/minicluster-data/postgres -l logfile start
2025-09-03 08:10:10.955 UTC [25642] LOG: starting PostgreSQL 17.2 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
2025-09-03 08:10:10.955 UTC [25642] LOG: listening on IPv4 address "127.12.201.212", port 49997
2025-09-03 08:10:10.959 UTC [25642] LOG: listening on Unix socket "/tmp/.s.PGSQL.49997"
2025-09-03 08:10:10.965 UTC [25647] LOG: database system was shut down at 2025-09-03 08:10:09 UTC
2025-09-03 08:10:10.972 UTC [25642] LOG: database system is ready to accept connections
I20250903 08:10:12.871057 13095 mini_postgres.cc:96] Postgres bound to 49997
2025-09-03 08:10:12.876 UTC [25653] FATAL: database "slave" does not exist
127.12.201.212:49997 - accepting connections
I20250903 08:10:12.877441 13095 mini_ranger.cc:161] Starting Ranger...
I20250903 08:10:12.894942 13095 mini_ranger.cc:84] Created miniranger Postgres user
I20250903 08:10:12.964866 13095 mini_ranger.cc:87] Created ranger Postgres database
I20250903 08:10:12.964947 13095 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/minicluster-data/ranger-admin
2025-09-03 08:10:13,268 [I] DB FLAVOR :POSTGRES
2025-09-03 08:10:13,268 [I] --------- Verifying Ranger DB connection ---------
2025-09-03 08:10:13,268 [I] Checking connection..
2025-09-03 08:10:13,268 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUML90r/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUML90r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.201.212:49997/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2025-09-03 08:10:13,583 [I] Checking connection passed.
2025-09-03 08:10:13,583 [I] --------- Verifying version history table ---------
2025-09-03 08:10:13,584 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUML90r/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUML90r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.201.212:49997/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2025-09-03 08:10:13,917 [I] Table x_db_version_h does not exist in database ranger
2025-09-03 08:10:13,917 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUML90r/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUML90r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.201.212:49997/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2025-09-03 08:10:14,241 [I] Table x_db_version_h does not exist in database ranger
2025-09-03 08:10:14,241 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2025-09-03 08:10:14,241 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUML90r/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUML90r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.201.212:49997/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskUML90r/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2025-09-03 08:10:14.554 UTC [25720] WARNING: there is no transaction in progress
2025-09-03 08:10:14,560 [I] create_dbversion_catalog.sql file imported successfully
2025-09-03 08:10:14,561 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUML90r/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUML90r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.201.212:49997/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2025-09-03 08:10:14,889 [I] Table x_db_version_h already exists in database 'ranger'
2025-09-03 08:10:14,889 [I] --------- Importing Ranger Core DB Schema ---------
2025-09-03 08:10:14,889 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUML90r/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUML90r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.201.212:49997/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2025-09-03 08:10:15,193 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUML90r/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUML90r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.201.212:49997/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2025-09-03 08:10:15,509 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUML90r/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUML90r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.201.212:49997/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-27r9.c.gcp-upstream.internal','N') ;"
2025-09-03 08:10:15,811 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUML90r/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUML90r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.201.212:49997/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2025-09-03 08:10:16,149 [I] Table x_portal_user does not exist in database ranger
2025-09-03 08:10:16,150 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUML90r/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUML90r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.201.212:49997/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2025-09-03 08:10:16,486 [I] Table x_policy_ref_group does not exist in database ranger
2025-09-03 08:10:16,486 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUML90r/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUML90r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.201.212:49997/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2025-09-03 08:10:16,799 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUML90r/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUML90r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.201.212:49997/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2025-09-03 08:10:17,110 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2025-09-03 08:10:17,111 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUML90r/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUML90r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.201.212:49997/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskUML90r/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2025-09-03 08:10:17.972 UTC [25837] WARNING: there is no transaction in progress
2025-09-03 08:10:17.987 UTC [25837] WARNING: there is no transaction in progress
2025-09-03 08:10:18.003 UTC [25837] WARNING: there is no transaction in progress
2025-09-03 08:10:18.017 UTC [25837] WARNING: there is no transaction in progress
2025-09-03 08:10:18.030 UTC [25837] WARNING: there is no transaction in progress
2025-09-03 08:10:18.042 UTC [25837] WARNING: there is no transaction in progress
2025-09-03 08:10:18.103 UTC [25837] WARNING: there is no transaction in progress
2025-09-03 08:10:18.109 UTC [25837] WARNING: there is no transaction in progress
2025-09-03 08:10:18.119 UTC [25837] WARNING: there is no transaction in progress
2025-09-03 08:10:18.128 UTC [25837] WARNING: there is no transaction in progress
2025-09-03 08:10:18.138 UTC [25837] WARNING: there is no transaction in progress
2025-09-03 08:10:18.148 UTC [25837] WARNING: there is no transaction in progress
2025-09-03 08:10:18.156 UTC [25837] WARNING: there is no transaction in progress
2025-09-03 08:10:18.161 UTC [25837] WARNING: there is no transaction in progress
2025-09-03 08:10:18.165 UTC [25837] WARNING: there is no transaction in progress
2025-09-03 08:10:18.168 UTC [25837] WARNING: there is no transaction in progress
2025-09-03 08:10:18,576 [I] ranger_core_db_postgres.sql file imported successfully
2025-09-03 08:10:18,576 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUML90r/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUML90r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.201.212:49997/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2025-09-03 08:10:18,900 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2025-09-03 08:10:18,901 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUML90r/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUML90r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.201.212:49997/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2025-09-03 08:10:19,227 [I] Table x_portal_user already exists in database 'ranger'
2025-09-03 08:10:19,227 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUML90r/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUML90r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.201.212:49997/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2025-09-03 08:10:19,566 [I] Table x_policy_ref_group already exists in database 'ranger'
2025-09-03 08:10:19,567 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUML90r/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUML90r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.201.212:49997/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2025-09-03 08:10:19,899 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUML90r/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUML90r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.201.212:49997/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2025-09-03 08:10:20,215 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUML90r/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUML90r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.201.212:49997/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-27r9.c.gcp-upstream.internal';"
2025-09-03 08:10:20,508 [I] CORE_DB_SCHEMA import status has been updated
2025-09-03 08:10:20,509 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUML90r/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUML90r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.201.212:49997/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2025-09-03 08:10:20,840 [I] DB_PATCHES have already been applied
I20250903 08:10:20.846247 13095 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/minicluster-data/ranger-admin:/tmp/dist-test-taskUML90r/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskUML90r/thirdparty/src/hadoop-3.4.1/*:/tmp/dist-test-taskUML90r/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUML90r/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20250903 08:10:20.846319 13095 mini_ranger.cc:193] Using host: 127.12.201.212
I20250903 08:10:20.849329 13095 mini_ranger.cc:227] Ranger admin URL: http://127.12.201.212:36769
Sep 03, 2025 8:10:21 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 03, 2025 8:10:21 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 03, 2025 8:10:21 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/minicluster-data/ranger-admin/ews/webapp
Sep 03, 2025 8:10:21 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 03, 2025 8:10:21 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 03, 2025 8:10:21 AM org.apache.catalina.core.StandardContext setPath
WARNING: A context path must either be an empty string or start with a '/' and do not end with a '/'. The path [/] does not meet these criteria and has been changed to []
Sep 03, 2025 8:10:21 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/minicluster-data/ranger-admin/ews/webapp].
Sep 03, 2025 8:10:21 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
Sep 03, 2025 8:10:21 AM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-bio-36769"]
Sep 03, 2025 8:10:21 AM org.apache.catalina.core.StandardService startInternal
INFO: Starting service [Tomcat]
Sep 03, 2025 8:10:21 AM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet Engine: Apache Tomcat/7.0.105
Sep 03, 2025 8:10:22 AM org.apache.catalina.loader.WebappClassLoaderBase validateJarFile
INFO: validateJarFile(/tmp/dist-test-taskUML90r/thirdparty/src/ranger-2.1.0-admin/ews/webapp/WEB-INF/lib/javax.servlet-api-3.1.0.jar) - jar not loaded. See Servlet Spec 3.0, section 10.7.2. Offending class: javax/servlet/Servlet.class
Sep 03, 2025 8:10:22 AM org.apache.catalina.loader.WebappClassLoaderBase validateJarFile
INFO: validateJarFile(/tmp/dist-test-taskUML90r/thirdparty/src/ranger-2.1.0-admin/ews/webapp/WEB-INF/lib/tomcat-el-api-7.0.105.jar) - jar not loaded. See Servlet Spec 3.0, section 10.7.2. Offending class: javax/el/Expression.class
Sep 03, 2025 8:10:22 AM org.apache.catalina.startup.ContextConfig getDefaultWebXmlFragment
INFO: No global web.xml found
Sep 03, 2025 8:10:22 AM org.apache.catalina.startup.TaglibUriRule body
INFO: TLD skipped. URI: urn:com:sun:jersey:api:view is already defined
Sep 03, 2025 8:10:22 AM org.apache.catalina.startup.TldConfig execute
INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
Sep 03, 2025 8:10:22 AM org.apache.catalina.core.ApplicationContext log
INFO: Set web app root system property: 'webapp.root' = [/tmp/dist-test-taskUML90r/thirdparty/src/ranger-2.1.0-admin/ews/webapp/]
Sep 03, 2025 8:10:22 AM org.apache.catalina.core.ApplicationContext log
INFO: Initializing Spring root WebApplicationContext
I20250903 08:10:23.093690 13095 mini_ranger.cc:160] Time spent starting Ranger: real 10.216s user 0.000s sys 0.009s
Sep 03, 2025 8:10:42 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 03, 2025 8:10:42 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 03, 2025 8:10:42 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 03, 2025 8:10:42 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 03, 2025 8:10:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 03, 2025 8:10:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 03, 2025 8:10:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 03, 2025 8:10:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 03, 2025 8:10:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 03, 2025 8:10:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 03, 2025 8:10:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 03, 2025 8:10:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 03, 2025 8:10:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 03, 2025 8:10:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 03, 2025 8:10:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 03, 2025 8:10:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 03, 2025 8:10:42 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 03, 2025 8:10:43 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
Sep 03, 2025 8:10:43 AM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["http-bio-36769"]
I20250903 08:10:44.146891 13095 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.12.201.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.12.201.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.12.201.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/krb5kdc/rangerkms_127.12.201.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.12.201.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/krb5kdc/rangerkms_127.12.201.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.12.201.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.12.201.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.201.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/krb5kdc/HTTP_127.12.201.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.12.201.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/krb5kdc/HTTP_127.12.201.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 03 08:10:44 dist-test-slave-27r9 krb5kdc[25611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756887044, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20250903 08:10:44.230212 13095 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20250903 08:10:44.239864 13095 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20250903 08:10:44.343209 13095 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20250903 08:10:44.343308 13095 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/minicluster-data/ranger-kms
I20250903 08:10:44.343330 13095 mini_ranger_kms.cc:227] Using postgres at 127.12.201.212:49997
2025-09-03 08:10:44,510 [I] DB FLAVOR :POSTGRES
2025-09-03 08:10:44,510 [I] --------- Verifying Ranger DB connection ---------
2025-09-03 08:10:44,510 [I] Checking connection
2025-09-03 08:10:44,510 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUML90r/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.201.212:49997/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2025-09-03 08:10:44,828 [I] connection success
2025-09-03 08:10:44,828 [I] --------- Verifying Ranger DB tables ---------
2025-09-03 08:10:44,828 [I] Verifying table ranger_masterkey in database rangerkms
2025-09-03 08:10:44,828 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUML90r/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.201.212:49997/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2025-09-03 08:10:45,134 [I] Table ranger_masterkey does not exist in database rangerkms
2025-09-03 08:10:45,134 [I] --------- Importing Ranger Core DB Schema ---------
2025-09-03 08:10:45,134 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2025-09-03 08:10:45,134 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUML90r/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.201.212:49997/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2025-09-03 08:10:45,435 [I] kms_core_db_postgres.sql DB schema imported successfully
I20250903 08:10:45.902467 13095 mini_ranger_kms.cc:310] Created kms service
I20250903 08:10:46.132246 13095 mini_ranger_kms.cc:326] Created kudu user
I20250903 08:10:46.172364 13095 mini_ranger_kms.cc:343] Created rangerkms user
Sep 03 08:10:46 dist-test-slave-27r9 krb5kdc[25611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756887044, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.12.201.212@KRBTEST.COM
I20250903 08:10:46.521775 13095 mini_ranger_kms.cc:384] Added ranger policy
I20250903 08:10:46.521922 13095 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/minicluster-data/ranger-kms:/tmp/dist-test-taskUML90r/build/debug/bin/postgresql.jar:/tmp/dist-test-taskUML90r/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskUML90r/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20250903 08:10:46.521943 13095 mini_ranger_kms.cc:240] Using host: 127.12.201.212
I20250903 08:10:46.524946 13095 mini_ranger_kms.cc:276] Ranger KMS PID: 26029
I20250903 08:10:46.525019 13095 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.12.201.212:39433
Sep 03, 2025 8:10:47 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 03, 2025 8:10:47 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 03, 2025 8:10:47 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 03, 2025 8:10:47 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 03, 2025 8:10:47 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/minicluster-data/ranger-kms/ews/webapp].
Sep 03, 2025 8:10:47 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 03, 2025 8:10:48 AM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-bio-39433"]
Sep 03, 2025 8:10:48 AM org.apache.catalina.core.StandardService startInternal
INFO: Starting service [Tomcat]
Sep 03, 2025 8:10:48 AM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet Engine: Apache Tomcat/7.0.105
Sep 03, 2025 8:10:48 AM org.apache.catalina.startup.ContextConfig getDefaultWebXmlFragment
INFO: No global web.xml found
Sep 03, 2025 8:10:48 AM org.apache.catalina.startup.TldConfig execute
INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
Sep 03 08:10:48 dist-test-slave-27r9 krb5kdc[25611](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1756887048, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.201.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 08:10:48.738245 13095 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.508s user 0.010s sys 0.140s
I20250903 08:10:48.738360 13095 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20250903 08:10:48.738428 13095 mini_ranger_kms.cc:401] 127.12.201.212:39433/kms/v1/keys
Sep 03 08:10:49 dist-test-slave-27r9 krb5kdc[25611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756887048, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.201.212@KRBTEST.COM for HTTP/127.12.201.212@KRBTEST.COM
Sep 03 08:10:49 dist-test-slave-27r9 krb5kdc[25611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756887048, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.201.212@KRBTEST.COM for HTTP/127.12.201.212@KRBTEST.COM
Sep 03 08:10:51 dist-test-slave-27r9 krb5kdc[25611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756887051, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.12.201.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.201.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.201.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.201.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.201.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.201.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.201.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.201.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/krb5kdc/kudu.keytab.
I20250903 08:10:51.667565 13095 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUML90r/build/debug/bin/kudu
/tmp/dist-test-taskUML90r/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.201.254:43565
--webserver_interface=127.12.201.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.201.212:36945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.201.254:43565
--ranger_config_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.201.212:39433/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 08:10:51.769857 26082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:10:51.770042 26082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:10:51.770090 26082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:10:51.773139 26082 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250903 08:10:51.773195 26082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 08:10:51.773217 26082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:10:51.773236 26082 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250903 08:10:51.773258 26082 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250903 08:10:51.777321 26082 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.201.212:36945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/minicluster-data/master-0/wal
--ranger_kms_url=127.12.201.212:39433/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.12.201.254:43565
--ranger_config_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.201.254:43565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.201.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26082
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 07:50:08 UTC on 5fd53c4cbb9d
build id 7874
I20250903 08:10:51.778259 26082 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:10:51.779057 26082 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:10:51.784538 26090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:10:51.784579 26087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:10:51.784621 26088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:10:51.784747 26082 server_base.cc:1047] running on GCE node
I20250903 08:10:51.785274 26082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:10:51.786116 26082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:10:51.787261 26082 hybrid_clock.cc:648] HybridClock initialized: now 1756887051787240 us; error 39 us; skew 500 ppm
Sep 03 08:10:51 dist-test-slave-27r9 krb5kdc[25611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756887051, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 08:10:51.789723 26082 init.cc:377] Logged in from keytab as kudu/127.12.201.254@KRBTEST.COM (short username kudu)
I20250903 08:10:51.790678 26082 webserver.cc:480] Webserver started at http://127.12.201.254:46821/ using document root <none> and password file <none>
I20250903 08:10:51.791219 26082 fs_manager.cc:362] Metadata directory not provided
I20250903 08:10:51.791268 26082 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:10:51.791420 26082 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
Sep 03 08:10:51 dist-test-slave-27r9 krb5kdc[25611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756887051, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.254@KRBTEST.COM for HTTP/127.12.201.212@KRBTEST.COM
I20250903 08:10:51.852234 26082 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/minicluster-data/master-0/data/instance:
uuid: "467d41bc26c442b495a80046bf3fe849"
format_stamp: "Formatted at 2025-09-03 08:10:51 on dist-test-slave-27r9"
server_key: "826b403c46247bf62f1e37cf82fa6624"
server_key_iv: "dc14e43c289111482c9e0a1b3804b21f"
server_key_version: "kuduclusterkey@0"
I20250903 08:10:51.852770 26082 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/minicluster-data/master-0/wal/instance:
uuid: "467d41bc26c442b495a80046bf3fe849"
format_stamp: "Formatted at 2025-09-03 08:10:51 on dist-test-slave-27r9"
server_key: "826b403c46247bf62f1e37cf82fa6624"
server_key_iv: "dc14e43c289111482c9e0a1b3804b21f"
server_key_version: "kuduclusterkey@0"
I20250903 08:10:51.856288 26082 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250903 08:10:51.873042 26097 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:10:51.874433 26082 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250903 08:10:51.874564 26082 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/minicluster-data/master-0/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/minicluster-data/master-0/wal
uuid: "467d41bc26c442b495a80046bf3fe849"
format_stamp: "Formatted at 2025-09-03 08:10:51 on dist-test-slave-27r9"
server_key: "826b403c46247bf62f1e37cf82fa6624"
server_key_iv: "dc14e43c289111482c9e0a1b3804b21f"
server_key_version: "kuduclusterkey@0"
I20250903 08:10:51.874694 26082 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:10:51.890920 26082 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:10:51.893901 26082 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:10:51.894081 26082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:10:51.902915 26082 rpc_server.cc:307] RPC server started. Bound to: 127.12.201.254:43565
I20250903 08:10:51.902933 26149 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.201.254:43565 every 8 connection(s)
I20250903 08:10:51.904007 26082 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/minicluster-data/master-0/data/info.pb
I20250903 08:10:51.906898 26150 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:10:51.913060 26150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 467d41bc26c442b495a80046bf3fe849: Bootstrap starting.
I20250903 08:10:51.914215 13095 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUML90r/build/debug/bin/kudu as pid 26082
I20250903 08:10:51.914323 13095 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/minicluster-data/master-0/wal/instance
I20250903 08:10:51.914543 13095 external_mini_cluster.cc:1468] Setting key a8416a166c0e51dc05341de5a8d04c0e
I20250903 08:10:51.915944 26150 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 467d41bc26c442b495a80046bf3fe849: Neither blocks nor log segments found. Creating new log.
I20250903 08:10:51.916796 26150 log.cc:826] T 00000000000000000000000000000000 P 467d41bc26c442b495a80046bf3fe849: Log is configured to *not* fsync() on all Append() calls
I20250903 08:10:51.918792 26150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 467d41bc26c442b495a80046bf3fe849: No bootstrap required, opened a new log
Sep 03 08:10:51 dist-test-slave-27r9 krb5kdc[25611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756887051, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.201.254@KRBTEST.COM
I20250903 08:10:51.922379 26150 raft_consensus.cc:357] T 00000000000000000000000000000000 P 467d41bc26c442b495a80046bf3fe849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "467d41bc26c442b495a80046bf3fe849" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 43565 } }
I20250903 08:10:51.922595 26150 raft_consensus.cc:383] T 00000000000000000000000000000000 P 467d41bc26c442b495a80046bf3fe849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:10:51.922689 26150 raft_consensus.cc:738] T 00000000000000000000000000000000 P 467d41bc26c442b495a80046bf3fe849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 467d41bc26c442b495a80046bf3fe849, State: Initialized, Role: FOLLOWER
I20250903 08:10:51.923182 26150 consensus_queue.cc:260] T 00000000000000000000000000000000 P 467d41bc26c442b495a80046bf3fe849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "467d41bc26c442b495a80046bf3fe849" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 43565 } }
I20250903 08:10:51.923328 26150 raft_consensus.cc:397] T 00000000000000000000000000000000 P 467d41bc26c442b495a80046bf3fe849 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250903 08:10:51.923411 26150 raft_consensus.cc:491] T 00000000000000000000000000000000 P 467d41bc26c442b495a80046bf3fe849 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250903 08:10:51.923524 26150 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 467d41bc26c442b495a80046bf3fe849 [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:10:51.924522 26150 raft_consensus.cc:513] T 00000000000000000000000000000000 P 467d41bc26c442b495a80046bf3fe849 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "467d41bc26c442b495a80046bf3fe849" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 43565 } }
I20250903 08:10:51.924887 26150 leader_election.cc:304] T 00000000000000000000000000000000 P 467d41bc26c442b495a80046bf3fe849 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 467d41bc26c442b495a80046bf3fe849; no voters:
I20250903 08:10:51.925262 26150 leader_election.cc:290] T 00000000000000000000000000000000 P 467d41bc26c442b495a80046bf3fe849 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250903 08:10:51.925662 26156 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 467d41bc26c442b495a80046bf3fe849 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 08:10:51.925873 26156 raft_consensus.cc:695] T 00000000000000000000000000000000 P 467d41bc26c442b495a80046bf3fe849 [term 1 LEADER]: Becoming Leader. State: Replica: 467d41bc26c442b495a80046bf3fe849, State: Running, Role: LEADER
I20250903 08:10:51.926183 26156 consensus_queue.cc:237] T 00000000000000000000000000000000 P 467d41bc26c442b495a80046bf3fe849 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "467d41bc26c442b495a80046bf3fe849" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 43565 } }
I20250903 08:10:51.926429 26153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:10:51.915575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33490 (local address 127.12.201.254:43565)
0903 08:10:51.916018 (+ 443us) server_negotiation.cc:207] Beginning negotiation
0903 08:10:51.916030 (+ 12us) server_negotiation.cc:400] Waiting for connection header
0903 08:10:51.916072 (+ 42us) server_negotiation.cc:408] Connection header received
0903 08:10:51.916595 (+ 523us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:10:51.916610 (+ 15us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:10:51.916902 (+ 292us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:10:51.917210 (+ 308us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:10:51.918105 (+ 895us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:10:51.918864 (+ 759us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:10:51.919550 (+ 686us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:10:51.919848 (+ 298us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:10:51.921914 (+ 2066us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:10:51.921933 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:10:51.921943 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:10:51.921972 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:10:51.923810 (+ 1838us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:10:51.924207 (+ 397us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:10:51.924211 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:10:51.924215 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:10:51.924274 (+ 59us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:10:51.924484 (+ 210us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:10:51.924488 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:10:51.924490 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:10:51.924845 (+ 355us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:10:51.925029 (+ 184us) server_negotiation.cc:1036] Waiting for connection context
0903 08:10:51.925300 (+ 271us) server_negotiation.cc:300] Negotiation successful
0903 08:10:51.925652 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":157,"threads_started":1}
I20250903 08:10:51.926566 26150 sys_catalog.cc:564] T 00000000000000000000000000000000 P 467d41bc26c442b495a80046bf3fe849 [sys.catalog]: configured and running, proceeding with master startup.
I20250903 08:10:51.927392 26157 sys_catalog.cc:455] T 00000000000000000000000000000000 P 467d41bc26c442b495a80046bf3fe849 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "467d41bc26c442b495a80046bf3fe849" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "467d41bc26c442b495a80046bf3fe849" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 43565 } } }
I20250903 08:10:51.927489 26157 sys_catalog.cc:458] T 00000000000000000000000000000000 P 467d41bc26c442b495a80046bf3fe849 [sys.catalog]: This master's current role is: LEADER
I20250903 08:10:51.927443 26155 sys_catalog.cc:455] T 00000000000000000000000000000000 P 467d41bc26c442b495a80046bf3fe849 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 467d41bc26c442b495a80046bf3fe849. Latest consensus state: current_term: 1 leader_uuid: "467d41bc26c442b495a80046bf3fe849" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "467d41bc26c442b495a80046bf3fe849" member_type: VOTER last_known_addr { host: "127.12.201.254" port: 43565 } } }
I20250903 08:10:51.927543 26155 sys_catalog.cc:458] T 00000000000000000000000000000000 P 467d41bc26c442b495a80046bf3fe849 [sys.catalog]: This master's current role is: LEADER
I20250903 08:10:51.929049 26150 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20250903 08:10:51.929857 26162 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250903 08:10:51.936848 26162 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250903 08:10:51.945144 26162 catalog_manager.cc:1349] Generated new cluster ID: 59ac82974ca64f01bd5516c332f4c765
I20250903 08:10:51.945328 26162 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250903 08:10:51.969470 26162 catalog_manager.cc:1372] Generated new certificate authority record
I20250903 08:10:51.970867 26162 catalog_manager.cc:1506] Loading token signing keys...
I20250903 08:10:51.987775 26162 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 467d41bc26c442b495a80046bf3fe849: Generated new TSK 0
I20250903 08:10:51.988734 26162 catalog_manager.cc:1516] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.201.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.201.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.201.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.201.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.201.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.201.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.201.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.201.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/krb5kdc/kudu.keytab.
I20250903 08:10:52.676368 13095 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUML90r/build/debug/bin/kudu
/tmp/dist-test-taskUML90r/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.201.212:39433/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.201.193:0
--local_ip_for_outbound_sockets=127.12.201.193
--webserver_interface=127.12.201.193
--webserver_port=0
--tserver_master_addrs=127.12.201.254:43565
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.201.212:36945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250903 08:10:52.786794 26196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:10:52.787034 26196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:10:52.787117 26196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:10:52.790247 26196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 08:10:52.790307 26196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:10:52.790417 26196 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.201.193
I20250903 08:10:52.794620 26196 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.201.212:36945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.12.201.212:39433/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.201.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.201.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.12.201.254:43565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26196
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.201.193
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 07:50:08 UTC on 5fd53c4cbb9d
build id 7874
I20250903 08:10:52.795651 26196 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:10:52.796504 26196 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:10:52.802762 26201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:10:52.803169 26204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:10:52.803383 26202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:10:52.803715 26196 server_base.cc:1047] running on GCE node
I20250903 08:10:52.804140 26196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:10:52.804788 26196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:10:52.808070 26196 hybrid_clock.cc:648] HybridClock initialized: now 1756887052808031 us; error 51 us; skew 500 ppm
Sep 03 08:10:52 dist-test-slave-27r9 krb5kdc[25611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756887052, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 08:10:52.811026 26196 init.cc:377] Logged in from keytab as kudu/127.12.201.193@KRBTEST.COM (short username kudu)
I20250903 08:10:52.812206 26196 webserver.cc:480] Webserver started at http://127.12.201.193:43967/ using document root <none> and password file <none>
I20250903 08:10:52.812773 26196 fs_manager.cc:362] Metadata directory not provided
I20250903 08:10:52.812829 26196 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:10:52.813017 26196 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
Sep 03 08:10:52 dist-test-slave-27r9 krb5kdc[25611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756887052, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.193@KRBTEST.COM for HTTP/127.12.201.212@KRBTEST.COM
I20250903 08:10:52.826053 26196 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/minicluster-data/ts-0/data/instance:
uuid: "e30eebb9271547188dbc984518a14b17"
format_stamp: "Formatted at 2025-09-03 08:10:52 on dist-test-slave-27r9"
server_key: "d7c3ab4524f6228257d65970baf990e0"
server_key_iv: "20f7419e1137df95f20e594bf70a63a3"
server_key_version: "kuduclusterkey@0"
I20250903 08:10:52.826735 26196 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/minicluster-data/ts-0/wal/instance:
uuid: "e30eebb9271547188dbc984518a14b17"
format_stamp: "Formatted at 2025-09-03 08:10:52 on dist-test-slave-27r9"
server_key: "d7c3ab4524f6228257d65970baf990e0"
server_key_iv: "20f7419e1137df95f20e594bf70a63a3"
server_key_version: "kuduclusterkey@0"
I20250903 08:10:52.831451 26196 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20250903 08:10:52.852030 26213 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:10:52.853518 26196 fs_manager.cc:730] Time spent opening block manager: real 0.007s user 0.003s sys 0.000s
I20250903 08:10:52.853682 26196 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/minicluster-data/ts-0/wal
uuid: "e30eebb9271547188dbc984518a14b17"
format_stamp: "Formatted at 2025-09-03 08:10:52 on dist-test-slave-27r9"
server_key: "d7c3ab4524f6228257d65970baf990e0"
server_key_iv: "20f7419e1137df95f20e594bf70a63a3"
server_key_version: "kuduclusterkey@0"
I20250903 08:10:52.853825 26196 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:10:52.868422 26196 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:10:52.872017 26196 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:10:52.872291 26196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:10:52.873015 26196 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 08:10:52.874218 26196 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 08:10:52.874330 26196 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 08:10:52.874461 26196 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 08:10:52.874542 26196 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 08:10:52.885507 26196 rpc_server.cc:307] RPC server started. Bound to: 127.12.201.193:34025
I20250903 08:10:52.886816 26196 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/minicluster-data/ts-0/data/info.pb
I20250903 08:10:52.894130 26326 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.201.193:34025 every 8 connection(s)
I20250903 08:10:52.894850 13095 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUML90r/build/debug/bin/kudu as pid 26196
I20250903 08:10:52.895002 13095 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/minicluster-data/ts-0/wal/instance
I20250903 08:10:52.895284 13095 external_mini_cluster.cc:1468] Setting key fde9816f0edc08a87dfc735a90d3baca
Sep 03 08:10:52 dist-test-slave-27r9 krb5kdc[25611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756887052, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.193@KRBTEST.COM for kudu/127.12.201.254@KRBTEST.COM
I20250903 08:10:52.909011 26331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:10:52.897229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.201.193:51447 (local address 127.12.201.254:43565)
0903 08:10:52.897593 (+ 364us) server_negotiation.cc:207] Beginning negotiation
0903 08:10:52.897600 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0903 08:10:52.898285 (+ 685us) server_negotiation.cc:408] Connection header received
0903 08:10:52.899271 (+ 986us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:10:52.899275 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:10:52.899329 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:10:52.899421 (+ 92us) server_negotiation.cc:227] Negotiated authn=SASL
0903 08:10:52.900926 (+ 1505us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:10:52.901733 (+ 807us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 08:10:52.902664 (+ 931us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 08:10:52.902881 (+ 217us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:10:52.905219 (+ 2338us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0903 08:10:52.905235 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0903 08:10:52.905238 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0903 08:10:52.905261 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0903 08:10:52.906732 (+ 1471us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:10:52.907338 (+ 606us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:10:52.907341 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:10:52.907343 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:10:52.907396 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0903 08:10:52.907731 (+ 335us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0903 08:10:52.907735 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0903 08:10:52.907737 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0903 08:10:52.907931 (+ 194us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0903 08:10:52.908016 (+ 85us) server_negotiation.cc:1036] Waiting for connection context
0903 08:10:52.908689 (+ 673us) server_negotiation.cc:300] Negotiation successful
0903 08:10:52.908836 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":110,"threads_started":1}
I20250903 08:10:52.910544 26332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:10:52.897613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:43565 (local address 127.12.201.193:51447)
0903 08:10:52.898125 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0903 08:10:52.898178 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0903 08:10:52.899085 (+ 907us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:10:52.899607 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0903 08:10:52.899621 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0903 08:10:52.899958 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0903 08:10:52.900745 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:10:52.900763 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:10:52.901895 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0903 08:10:52.901898 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0903 08:10:52.902505 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0903 08:10:52.902516 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0903 08:10:52.903209 (+ 693us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0903 08:10:52.903826 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0903 08:10:52.903851 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0903 08:10:52.905072 (+ 1221us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0903 08:10:52.906879 (+ 1807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:10:52.906885 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:10:52.906899 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:10:52.907218 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:10:52.907502 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0903 08:10:52.907505 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0903 08:10:52.907507 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0903 08:10:52.907623 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0903 08:10:52.908086 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0903 08:10:52.908092 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0903 08:10:52.908382 (+ 290us) client_negotiation.cc:769] Sending connection context
0903 08:10:52.909471 (+ 1089us) client_negotiation.cc:240] Negotiation successful
0903 08:10:52.909732 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":110,"threads_started":1}
I20250903 08:10:52.912206 26327 heartbeater.cc:344] Connected to a master server at 127.12.201.254:43565
I20250903 08:10:52.912489 26327 heartbeater.cc:461] Registering TS with master...
I20250903 08:10:52.913141 26327 heartbeater.cc:507] Master 127.12.201.254:43565 requested a full tablet report, sending...
I20250903 08:10:52.914769 26111 ts_manager.cc:194] Registered new tserver with Master: e30eebb9271547188dbc984518a14b17 (127.12.201.193:34025)
I20250903 08:10:52.916378 26111 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.201.193@KRBTEST.COM'} at 127.12.201.193:51447
WARNING: no policy specified for kudu/127.12.201.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.201.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.201.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.201.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.201.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.201.194@KRBTEST.COM" created.
Exception in thread "main" org.apache.kudu.subprocess.KuduSubprocessException: Failed to login with Kudu principal/keytab
at org.apache.kudu.subprocess.ranger.authorization.RangerKuduAuthorizer.init(RangerKuduAuthorizer.java:87)
at org.apache.kudu.subprocess.ranger.RangerProtocolHandler.<init>(RangerProtocolHandler.java:45)
at org.apache.kudu.subprocess.ranger.RangerSubprocessMain.main(RangerSubprocessMain.java:39)
Caused by: org.apache.hadoop.security.KerberosAuthException: failure to login: for principal: kudu/127.12.201.254 from keytab /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/krb5kdc/kudu.keytab javax.security.auth.login.LoginException: Unable to obtain password from user
at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:2064)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:1398)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:1136)
at org.apache.kudu.subprocess.ranger.authorization.RangerKuduAuthorizer.init(RangerKuduAuthorizer.java:85)
... 2 more
Caused by: javax.security.auth.login.LoginException: Unable to obtain password from user
at com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:901)
at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:764)
at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:618)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
at org.apache.hadoop.security.UserGroupInformation$HadoopLoginContext.login(UserGroupInformation.java:2148)
at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:2053)
... 5 more
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.201.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.201.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/krb5kdc/kudu.keytab.
I20250903 08:10:52.965541 13095 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUML90r/build/debug/bin/kudu
/tmp/dist-test-taskUML90r/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.201.212:39433/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.201.194:0
--local_ip_for_outbound_sockets=127.12.201.194
--webserver_interface=127.12.201.194
--webserver_port=0
--tserver_master_addrs=127.12.201.254:43565
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.201.212:36945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20250903 08:10:52.983328 26188 server.cc:273] Received an EOF from the subprocess
F20250903 08:10:52.984285 26175 server.cc:406] The subprocess has exited with status 1
*** Check failure stack trace: ***
@ 0x7fa56c7cddcd google::LogMessage::Fail() at ??:0
@ 0x7fa56c7d1b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fa56c7cd7cc google::LogMessage::Flush() at ??:0
@ 0x7fa56c7cef59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fa56e3d92d7 kudu::subprocess::SubprocessServer::ExitCheckerThread() at ??:0
@ 0x7fa56e3d60d9 _ZZN4kudu10subprocess16SubprocessServer4InitEvENKUlvE0_clEv at ??:0
@ 0x7fa56e3da005 _ZNSt17_Function_handlerIFvvEZN4kudu10subprocess16SubprocessServer4InitEvEUlvE0_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55eee9ae2364 std::function<>::operator()() at ??:0
@ 0x7fa56dc23924 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fa56e4156db start_thread at ??:0
@ 0x7fa56b98471f clone at ??:0
W20250903 08:10:53.093485 26338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:10:53.093695 26338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:10:53.093763 26338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:10:53.097007 26338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250903 08:10:53.097069 26338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:10:53.097157 26338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.201.194
I20250903 08:10:53.101306 26338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.201.212:36945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.12.201.212:39433/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.201.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.201.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.201.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.12.201.254:43565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26338
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.201.194
--log_dir=/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type DEBUG
built by None at 03 Sep 2025 07:50:08 UTC on 5fd53c4cbb9d
build id 7874
I20250903 08:10:53.102340 26338 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:10:53.103224 26338 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:10:53.109447 26354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:10:53.109583 26355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:10:53.109440 26357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:10:53.109815 26338 server_base.cc:1047] running on GCE node
I20250903 08:10:53.110220 26338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:10:53.110843 26338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:10:53.112030 26338 hybrid_clock.cc:648] HybridClock initialized: now 1756887053112028 us; error 44 us; skew 500 ppm
Sep 03 08:10:53 dist-test-slave-27r9 krb5kdc[25611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756887053, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250903 08:10:53.115334 26338 init.cc:377] Logged in from keytab as kudu/127.12.201.194@KRBTEST.COM (short username kudu)
I20250903 08:10:53.116590 26338 webserver.cc:480] Webserver started at http://127.12.201.194:35399/ using document root <none> and password file <none>
I20250903 08:10:53.117300 26338 fs_manager.cc:362] Metadata directory not provided
I20250903 08:10:53.117369 26338 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:10:53.117599 26338 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
Sep 03 08:10:53 dist-test-slave-27r9 krb5kdc[25611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756887053, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.201.194@KRBTEST.COM for HTTP/127.12.201.212@KRBTEST.COM
I20250903 08:10:53.132112 26338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/minicluster-data/ts-1/data/instance:
uuid: "a06e1b51f1c248e680ed52e9bee32371"
format_stamp: "Formatted at 2025-09-03 08:10:53 on dist-test-slave-27r9"
server_key: "8671cf082e2249a5dc91a1fe13fe7e05"
server_key_iv: "b4e24cbd2529f69d335829895332e14e"
server_key_version: "kuduclusterkey@0"
I20250903 08:10:53.133090 26338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/minicluster-data/ts-1/wal/instance:
uuid: "a06e1b51f1c248e680ed52e9bee32371"
format_stamp: "Formatted at 2025-09-03 08:10:53 on dist-test-slave-27r9"
server_key: "8671cf082e2249a5dc91a1fe13fe7e05"
server_key_iv: "b4e24cbd2529f69d335829895332e14e"
server_key_version: "kuduclusterkey@0"
I20250903 08:10:53.140023 26338 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.003s sys 0.000s
I20250903 08:10:53.177060 26364 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:10:53.178447 26338 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250903 08:10:53.178591 26338 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/minicluster-data/ts-1/wal
uuid: "a06e1b51f1c248e680ed52e9bee32371"
format_stamp: "Formatted at 2025-09-03 08:10:53 on dist-test-slave-27r9"
server_key: "8671cf082e2249a5dc91a1fe13fe7e05"
server_key_iv: "b4e24cbd2529f69d335829895332e14e"
server_key_version: "kuduclusterkey@0"
I20250903 08:10:53.178736 26338 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:10:53.204290 26338 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:10:53.207827 26338 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:10:53.208019 26338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:10:53.208662 26338 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 08:10:53.209700 26338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 08:10:53.209771 26338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 08:10:53.209836 26338 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 08:10:53.209867 26338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 08:10:53.220055 26338 rpc_server.cc:307] RPC server started. Bound to: 127.12.201.194:33217
I20250903 08:10:53.220082 26477 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.201.194:33217 every 8 connection(s)
I20250903 08:10:53.220983 26338 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/minicluster-data/ts-1/data/info.pb
I20250903 08:10:53.222072 13095 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUML90r/build/debug/bin/kudu as pid 26338
I20250903 08:10:53.222185 13095 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUML90r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1756886860253814-13095-0/minicluster-data/ts-1/wal/instance
I20250903 08:10:53.222429 13095 external_mini_cluster.cc:1468] Setting key ac5be5220408638ff6bb8bd439d4542f
W20250903 08:10:53.224895 25609 connection.cc:537] client connection to 127.12.201.254:43565 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250903 08:10:53.226058 26480 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 08:10:53.223642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:43565 (local address 127.12.201.194:53049)
0903 08:10:53.224323 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0903 08:10:53.224371 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0903 08:10:53.225296 (+ 925us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:10:53.225529 (+ 233us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.201.254:43565: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":130,"threads_started":1}
W20250903 08:10:53.226475 26478 heartbeater.cc:646] Failed to heartbeat to 127.12.201.254:43565 (0 consecutive failures): Network error: Failed to ping master at 127.12.201.254:43565: Client connection negotiation failed: client connection to 127.12.201.254:43565: BlockingWrite error: write error: Connection reset by peer (error 104)
I20250903 08:10:53.227080 26480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:10:53.226842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:43565 (local address 127.12.201.194:41571)
0903 08:10:53.226950 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0903 08:10:53.227025 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.201.254:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20250903 08:10:53.227667 26480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:10:53.227474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:43565 (local address 127.12.201.194:46313)
0903 08:10:53.227574 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0903 08:10:53.227611 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.201.254:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20250903 08:10:53.227835 26478 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20250903 08:10:53.919225 26327 heartbeater.cc:499] Master 127.12.201.254:43565 was elected leader, sending a full tablet report...
I20250903 08:10:53.920128 26482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:10:53.919705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:43565 (local address 127.12.201.193:58221)
0903 08:10:53.919955 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0903 08:10:53.920021 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.201.254:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":75,"threads_started":1}
W20250903 08:10:53.920397 26327 heartbeater.cc:646] Failed to heartbeat to 127.12.201.254:43565 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.12.201.254:43565: connect: Connection refused (error 111)
I20250903 08:10:54.228772 26483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:10:54.228381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:43565 (local address 127.12.201.194:38593)
0903 08:10:54.228618 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0903 08:10:54.228669 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.201.254:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":66,"threads_started":1}
I20250903 08:10:54.921569 26484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:10:54.921144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:43565 (local address 127.12.201.193:60175)
0903 08:10:54.921407 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0903 08:10:54.921472 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.201.254:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":75,"threads_started":1}
I20250903 08:10:55.229960 26485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:10:55.229525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:43565 (local address 127.12.201.194:60665)
0903 08:10:55.229795 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0903 08:10:55.229869 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.201.254:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":71,"threads_started":1}
I20250903 08:10:55.922716 26486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:10:55.922275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:43565 (local address 127.12.201.193:38551)
0903 08:10:55.922559 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0903 08:10:55.922618 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.201.254:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":90,"threads_started":1}
I20250903 08:10:56.231124 26487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:10:56.230673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:43565 (local address 127.12.201.194:57831)
0903 08:10:56.230937 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0903 08:10:56.231025 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.201.254:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":82,"threads_started":1}
I20250903 08:10:56.924170 26488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:10:56.923653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:43565 (local address 127.12.201.193:36841)
0903 08:10:56.924005 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0903 08:10:56.924066 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.201.254:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":110,"threads_started":1}
I20250903 08:10:57.232391 26489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:10:57.231918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:43565 (local address 127.12.201.194:42911)
0903 08:10:57.232218 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0903 08:10:57.232282 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.201.254:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":90,"threads_started":1}
I20250903 08:10:57.925549 26490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:10:57.925069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:43565 (local address 127.12.201.193:59221)
0903 08:10:57.925364 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0903 08:10:57.925422 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.201.254:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":90,"threads_started":1}
I20250903 08:10:58.233750 26491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0903 08:10:58.233288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.201.254:43565 (local address 127.12.201.194:40055)
0903 08:10:58.233599 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0903 08:10:58.233652 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.201.254:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":100,"threads_started":1}
F20250903 08:10:58.262190 13095 external_mini_cluster.cc:650] Check failed: leader_master() != nullptr Must have started at least 1 master before adding tablet servers
*** Check failure stack trace: ***
*** Aborted at 1756887058 (unix time) try "date -d @1756887058" if you are using GNU date ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e800003327) received by PID 13095 (TID 0x7f5817ae6d80) from PID 13095; stack trace: ***
@ 0x7f5823608980 (unknown) at ??:0
@ 0x7f5820ceafb7 gsignal at ??:0
@ 0x7f5820cec921 abort at ??:0
@ 0x7f5821f58dcd google::LogMessage::Fail() at ??:0
@ 0x7f5821f5cb93 google::LogMessage::SendToLog() at ??:0
@ 0x7f5821f587cc google::LogMessage::Flush() at ??:0
@ 0x7f5821f59f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f5824bd9cab kudu::cluster::ExternalMiniCluster::AddTabletServer() at ??:0
@ 0x7f5824bd69c6 kudu::cluster::ExternalMiniCluster::Start() at ??:0
@ 0x561fe0200919 kudu::SecurityITest::StartCluster() at ??:0
@ 0x561fe01e8c15 kudu::SecurityITest_TestEncryptionWithKMSIntegrationMultipleServers_Test::TestBody() at ??:0
@ 0x7f5822f630ed testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
@ 0x7f5822f57bdb testing::Test::Run() at ??:0
@ 0x7f5822f57d9d testing::TestInfo::Run() at ??:0
@ 0x7f5822f58377 testing::TestSuite::Run() at ??:0
@ 0x7f5822f5877c testing::internal::UnitTestImpl::RunAllTests() at ??:0
@ 0x7f5822f6360d testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
@ 0x7f5822f57e63 testing::UnitTest::Run() at ??:0
@ 0x561fe0238c58 RUN_ALL_TESTS() at ??:0
@ 0x561fe0237e45 main at ??:0
@ 0x7f5820ccdbf7 __libc_start_main at ??:0
@ 0x561fe01d006a _start at ??:0
2025-09-03 08:10:58.447 UTC [26060] FATAL: terminating connection due to unexpected postmaster exit
2025-09-03 08:10:58.448 UTC [25956] FATAL: terminating connection due to unexpected postmaster exit
2025-09-03 08:10:58.448 UTC [25957] FATAL: terminating connection due to unexpected postmaster exit
2025-09-03 08:10:58.450 UTC [25953] FATAL: terminating connection due to unexpected postmaster exit
2025-09-03 08:10:58.450 UTC [25954] FATAL: terminating connection due to unexpected postmaster exit
2025-09-03 08:10:58.451 UTC [25952] FATAL: terminating connection due to unexpected postmaster exit